Bug 1278637

Summary: Support for encrypted Hyper-V connection in virt-who
Product: Red Hat Enterprise Linux 7 Reporter: Anand Vaddarapu <avaddara>
Component: virt-whoAssignee: Radek Novacek <rnovacek>
Status: CLOSED ERRATA QA Contact: Eko <hsun>
Severity: urgent Docs Contact: Yehuda Zimmerman <yzimmerm>
Priority: urgent    
Version: 7.1CC: ahumbe, alanm, avaddara, cww, jdeenada, ldai, liliu, mnapolis, ovasik, rnovacek, sgao, shihliu, xdmoon, yzimmerm
Target Milestone: rc   
Target Release: 7.1   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Encrypted Hyper-V connections supported in _virt-who_ Previously, _virt-who_ used unencrypted Hyper-V connections. All data was sent in plain text. This had security implications and needed special configuration on Hyper-V servers to be allowed. With this update, _virt-who_ now uses Windows NT LAN Manager (NTLM) sealing and signing to protect communication with Hyper-V servers.
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 05:06:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1291737    
Bug Blocks: 1172231, 1203710, 1272873, 1298337, 1313485    
Attachments:
Description Flags
Disallowed auth with GPO.png
none
Disallowed auth without GPO.png
none
Allow auth with GPO.png none

Description Anand Vaddarapu 2015-11-06 04:02:13 UTC
Description of problem:

virt-who configs:

#cat /etc/virt-who.d/hyperv_config
[hypervname]
type=hyperv
#server=hyperV-FQDN
server=http://10.0.0.0
username=username
encrypted_password=<encrypted using virt-who-password>
owner=OWNER
env=GA-Static-Patching-RHEL-7
hypervisor_id=hostname

>winrm get winrm/config/service
Service
    RootSDDL = XXXXX
    MaxConcurrentOperations = 4294967295
    MaxConcurrentOperationsPerUser = 1500
    EnumerationTimeoutms = 240000
    MaxConnections = 300
    MaxPacketRetrievalTimeSeconds = 120
    AllowUnencrypted = true [Source="GPO"]
    Auth
        Basic = false [Source="GPO"]
        Kerberos = true
        Negotiate = true [Source="GPO"]
        Certificate = false
        CredSSP = true
        CbtHardeningLevel = Relaxed
    DefaultPorts
        HTTP = 5985
        HTTPS = 5986
    IPv4Filter = * [Source="GPO"]
    IPv6Filter = * [Source="GPO"]
    EnableCompatibilityHttpListener = true
    EnableCompatibilityHttpsListener = true
    CertificateThumbprint
    AllowRemoteAccess = true [Source="GPO"]

virt-who -o -d
ERROR:root:Configuration file katello-ca-consumer-latest.noarch.rpm contains no section headers
2015-11-05 15:33:57,762 DEBUG: Using config named 'hyperv-hostname1'
2015-11-05 15:33:57,762 DEBUG: Using config named 'hyperv-hostname'
2015-11-05 15:33:57,763 INFO: Using configuration "hyperv-hostname1" ("hyperv" mode)
2015-11-05 15:33:57,763 INFO: Using configuration "hyperv-hostname2" ("hyperv" mode)
2015-11-05 15:33:57,829 DEBUG: Hyper-V url: http://10.0.0.0:5985/wsman
2015-11-05 15:33:57,833 DEBUG: Hyper-V url: http://10.0.0.0:5985/wsman
2015-11-05 15:33:57,841 DEBUG: Using NTLM authentication
2015-11-05 15:33:57,843 DEBUG: Using NTLM authentication
2015-11-05 15:33:57,887 ERROR: Virt backend 'hyperv-hostname1' fails with error: NTLM negotiation failed
2015-11-05 15:33:57,890 ERROR: Virt backend 'hyperv-hostname2' fails with error: NTLM negotiation failed

Version-Release number of selected component (if applicable):


How reproducible:
1. Install laste virt-who virt-who-0.14-4.el7.noarch.rpm
2. Configure virt-who with above configs
3. run virt-who -o -d to check autentication

Actual results:
ERROR: Virt backend 'hyperv-hostname2' fails with error: NTLM negotiation failed

Expected results:
Authentication should succeed.

Additional info:
Hyper-V are part of GPO (Microsoft Group Policy objects). When customer removes servers from GPO, virt-who can authenticate successfully.

Comment 1 Liushihui 2015-11-06 04:38:16 UTC
Please notice that: when Hyper-V is not part of GPO policy, virt-who can authenticate successfully only after set "AllowUnencrypted = false", However, if set "AllowUnencrypted = true",it still show " NTLM negotiation failed"

Comment 3 Radek Novacek 2015-11-24 07:34:30 UTC
Please, can you tell the customer to check if following setting in the Group Policy is set to "Disabled"

Computer > Policies > Administrative Templates > Windows Components > Windows Remote Management > WinRM Service: Disallow Negotiate Authentication.

Thanks.

Comment 7 Radek Novacek 2015-12-02 08:07:02 UTC
Ok, thanks. Did the customer try suggestion from comment #3?

Comment 10 Radek Novacek 2015-12-08 07:09:58 UTC
Jay,

looking at that screenshot, there is "Allow unencrypted traffic" option right above "Disallow Negotiate Authentication" and it is set to Disabled. virt-who is using unencrypted communication to talk to Hyper-V. Can the customer try to allow it?

Comment 13 Radek Novacek 2015-12-10 07:40:11 UTC
Anand,

can you please clarify what config options need to be changed in order to make virt-who work? I can work on finding some workaround but I need to know precisely what options in their GPO policy are not compatible with current virt-who requirements.

Comment 14 Radek Novacek 2015-12-10 07:47:17 UTC
Could the "AllowUnencrypted" option be the problem? virt-who needs this to be set to "true". In the bug description, this options is "true". In comment #4, there is "AllowUnencrypted = true [Source="GPO"]", but in comment #11, there is "AllowUnencrypted = false [Source="GPO"]"

Which one is customers original GPO setting?

Comment 16 Eko 2015-12-10 09:51:28 UTC
Hi Radek, I have set up an environment to reproduce this issue, my test result as following:
if set AllowUnencrypted = false [Source="GPO"], virt-who Failed;
if set AllowUnencrypted = true [Source="GPO"], virt-who pass;
if set AllowUnencrypted = true, virt-who pass;

so maybe it's not a GPO issue, the problem is how to make virt-who can work normally with AllowUnencrypted = false?

Comment 17 Radek Novacek 2015-12-10 11:54:43 UTC
Thanks, I'll take a look what would it need to support encrypted Hyper-V connection in virt-who.

Comment 21 Radek Novacek 2015-12-22 07:06:28 UTC
Anand, 

the workaround it to enable unencrypted connections. I'll give it a try to implement it in January. If I succeed, we can do a z-stream fix for this issue.

Comment 30 Liushihui 2016-02-02 03:56:11 UTC
In the latest In the latest virt-who-0.16-2.el6.noarch. test with the following three GPS settings, virt-who can get through NTLM authentication and send host/guest mapping to server successfully.

1. set AllowUnencrypted = false [Source="GPO"], virt-who pass;see the screenshot "Disallowed auth with GPO.png"
 "AllowUnencrypted = false [Source="GPO"]
    Auth
        Basic = false [Source="GPO"]
        Negotiate = true [Source="GPO"]
        IPv4Filter = * [Source="GPO"]
        IPv6Filter = * [Source="GPO"]
        AllowRemoteAccess = true [Source="GPO"] "

2. set AllowUnencrypted = false, without GPO, virt-who pass;see the screenshot "Disallowed auth without GPO.png"
 "AllowUnencrypted = false
    Auth
        Basic = false [Source="GPO"]
        Negotiate = true
        IPv4Filter = * [Source="GPO"]
        IPv6Filter = * [Source="GPO"]
        AllowRemoteAccess = true [Source="GPO"] "

3. set AllowUnencrypted = true [Source="GPO"], virt-who pass; see the screenshot "allow auth with GPO.png"
AllowUnencrypted = true [Source="GPO"]
    Auth
        Basic = false [Source="GPO"]
        Kerberos = true
        Negotiate = true [Source="GPO"]
    IPv4Filter = * [Source="GPO"]
    IPv6Filter = * [Source="GPO"]
    AllowRemoteAccess = true [Source="GPO"]

Comment 31 Liushihui 2016-02-02 03:57:33 UTC
Created attachment 1120307 [details]
Disallowed auth with GPO.png

Comment 32 Liushihui 2016-02-02 03:59:10 UTC
Created attachment 1120308 [details]
Disallowed auth without GPO.png

Comment 33 Liushihui 2016-02-02 03:59:49 UTC
Created attachment 1120309 [details]
Allow auth with GPO.png

Comment 37 Radek Novacek 2016-02-23 08:19:46 UTC
Anand, we can't just release it. We have to follow the process. That means to follow a z-stream process. But I don't think this bug qualifies for z-stream. This is clearly a new feature request. z-stream is meant to be for critical bug fixes only.

There is a risk that this change will break deployments for existing customers because it substantially changes how virt-who connects to hyper-v. I think we should wait for RHEL 7.3 GA.

If the customer really needs to have this ASAP, feel free to investigate options we have (z-stream, hotfix, async, etc.).

Comment 42 Radek Novacek 2016-03-08 06:50:15 UTC
The encrypted connection is already implemented upstream and will be resolved by rebase in 7.3. I will do the hotfix process ASAP.

Comment 44 Radek Novacek 2016-03-08 14:03:06 UTC
Hotfix build done: https://brewweb.devel.redhat.com/taskinfo?taskID=10620147

Comment 46 Radek Novacek 2016-03-09 07:44:57 UTC
Yes, although I would prefer QE team to test it first.

Comment 47 Liushihui 2016-03-10 04:38:07 UTC
Verified it on virt-who-0.14-9.el7.0.0.hotfix.1.bz1278637.noarch since virt-who can get through NTLM authentication and send host/guest mapping to server successfully when set "AllowUnencrypted = false" or "AllowUnencrypted = true".Meanwhile, guest can subscribe bonus pool successfully after hypervisor subscribe physical pool.

Checked version:
virt-who-0.14-9.el7.0.0.hotfix.1.bz1278637.noarch
subscription-manager-1.15.9-15.el7.x86_64
python-rhsm-1.15.4-5.el7.x86_64

Checked process:
1. Update virt-who version to the hostfix version
[root@hp-xl220agen8v2-01 ~]# rpm -q virt-who
virt-who-0.14-9.el7.noarch
[root@hp-xl220agen8v2-01 ~]# yum install -y python-requests
[root@hp-xl220agen8v2-01 ~]# rpm -Uvh virt-who-0.14-9.el7.0.0.hotfix.1.bz1278637.noarch.rpm 
Preparing...                          ################################# [100%]
Updating / installing...
   1:virt-who-0.14-9.el7.0.0.hotfix.1.################################# [ 50%]
Cleaning up / removing...
   2:virt-who-0.14-9.el7              ################################# [100%]
[root@hp-xl220agen8v2-01 ~]# rpm -q virt-who
virt-who-0.14-9.el7.0.0.hotfix.1.bz1278637.noarch
2. Register system to satellite6.1
3. In hyperv, configure local group policy as the following three conditions.
Config1: set AllowUnencrypted = false [Source="GPO"], virt-who pass;see the screenshot "Disallowed auth with GPO.png"
 "AllowUnencrypted = false [Source="GPO"]
    Auth
        Basic = false [Source="GPO"]
        Negotiate = true [Source="GPO"]
        IPv4Filter = * [Source="GPO"]
        IPv6Filter = * [Source="GPO"]
        AllowRemoteAccess = true [Source="GPO"] "
Config2. set AllowUnencrypted = false, without GPO, virt-who pass;see the screenshot "Disallowed auth without GPO.png"
 "AllowUnencrypted = false
    Auth
        Basic = false [Source="GPO"]
        Negotiate = true
        IPv4Filter = * [Source="GPO"]
        IPv6Filter = * [Source="GPO"]
        AllowRemoteAccess = true [Source="GPO"] "
Config3. set AllowUnencrypted = true [Source="GPO"], virt-who pass; see the screenshot "allow auth with GPO.png"
AllowUnencrypted = true [Source="GPO"]
    Auth
        Basic = false [Source="GPO"]
        Kerberos = true
        Negotiate = true [Source="GPO"]
    IPv4Filter = * [Source="GPO"]
    IPv6Filter = * [Source="GPO"]
    AllowRemoteAccess = true [Source="GPO"]
4. Configure virt-who run at hyperv mode ,restart virt-who and check virt-who's log
[root@hp-xl220agen8v2-01 ~]# cat /etc/virt-who.d/virt 
[test-hyperv1]
type=hyperv
server=10.73.5.227
username=administrator
password=Welcome1
owner=ACME_Corporation
env=Library 
[root@hp-xl220agen8v2-01 ~]# service virt-who restart && tail -f /var/log/rhsm/rhsm.log
2016-03-09 23:32:37,320 [INFO]  @virtwho.py:697 - Using configuration "test-hyperv1" ("hyperv" mode)
2016-03-09 23:32:37,320 [DEBUG]  @virtwho.py:216 - Starting infinite loop with 5 seconds interval
2016-03-09 23:32:37,358 [DEBUG]  @hyperv.py:477 - Hyper-V url: http://10.73.5.227:5985/wsman
2016-03-09 23:32:38,660 [DEBUG]  @hyperv.py:71 - Using NTLM authentication
2016-03-09 23:32:39,979 [DEBUG]  @hyperv.py:84 - Sending NTLM authentication data
2016-03-09 23:32:40,585 [DEBUG]  @hyperv.py:107 - NTLM authentication successful
2016-03-09 23:32:40,590 [DEBUG]  @hyperv.py:511 - Unable to enumerate using root/virtualization namespace, trying root/virtualization/v2 namespace
2016-03-09 23:32:44,461 [DEBUG]  @virt.py:343 - Getting the host/guests association took too long, interval waiting is skipped
2016-03-09 23:32:44,463 [DEBUG]  @subscriptionmanager.py:112 - Authenticating with certificate: /etc/pki/consumer/cert.pem
2016-03-09 23:32:44,589 [DEBUG]  @subscriptionmanager.py:146 - Checking if server has capability 'hypervisor_async'
2016-03-09 23:32:44,709 [DEBUG]  @subscriptionmanager.py:158 - Server does not have 'hypervisors_async' capability
2016-03-09 23:32:44,710 [INFO]  @subscriptionmanager.py:165 - Sending update in hosts-to-guests mapping: {
    "hyperv_01": [
        {
            "guestId": "32710A7E-94A9-A445-944E-16C01BFA63B3", 
            "state": 1, 
            "attributes": {
                "active": 1, 
                "virtWhoType": "hyperv", 
                "hypervisorType": "hyperv"
            }
        }, 
        {
            "guestId": "0E32F0E5-05CA-014A-BD59-F63D75843D5D", 
            "state": 1, 
            "attributes": {
                "active": 1, 
                "virtWhoType": "hyperv", 
                "hypervisorType": "hyperv"
            }
        }
    ]
}

5. In satellite webUI, Go to "content host" --> choose the [hyperv_hostnae]-->"Subscriptions" --> "Add", choose physical pool which can generate bonus pool on hypervisor ,then subscribe it.
6. In the Guest, list the bonus pool and subscribe the bonus pool

Result:
Virt-who send correct host/guest mapping info to satellite, guest can subscribe bonus pool successfully.

Comment 49 Radek Novacek 2016-03-10 06:41:22 UTC
Anand, yes, you can provide the package to the customer.

Comment 50 Eko 2016-04-27 05:40:35 UTC
*** Bug 1167283 has been marked as a duplicate of this bug. ***

Comment 51 Eko 2016-04-27 05:42:49 UTC
Verified it on virt-who-0.14-9.el7.0.0.hotfix.1.bz1278637.noarch,
and can't reproduce it in in virt-who-0.16-8.el6.noarch

Comment 53 Radek Novacek 2016-09-29 14:36:22 UTC
The updated Doc Text is fine. Thanks.

Comment 55 errata-xmlrpc 2016-11-04 05:06:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2387.html