Bug 1486270

Summary: [RFE] Add no_proxy setting in /etc/virt-who.conf
Product: Red Hat Enterprise Linux 7 Reporter: Kenny Tordeurs <ktordeur>
Component: virt-whoAssignee: William Poteat <wpoteat>
Status: CLOSED ERRATA QA Contact: Eko <hsun>
Severity: high Docs Contact:
Priority: high    
Version: 7.4CC: csnyder, hasuzuki, ktordeur, rjerrido, shughes, smutkule, vgunasek, vmeghana, wpoteat, yuefliu
Target Milestone: rcKeywords: FutureFeature
Target Release: 7.7   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-06 12:40:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1647936    

Description Kenny Tordeurs 2017-08-29 10:24:06 UTC 
Description of problem:
NO_PROXY=* is being ignored when it should be working as described below according to `man virt-who-config`
~~~
VARIABLES UNIQUE TO SYSCONFIG
NO_PROXY
A  comma-separated  list  of  hostnames  or  domains  or ip addresses to ignore proxy settings for.  Optionally this may be set to '*' to bypass proxy settings for all hostnames domains or ip
addresses.
~~~

Errata installed as mentioned in bz 1311429 https://access.redhat.com/errata/RHBA-2017:2083
Seems a regression from https://bugzilla.redhat.com/show_bug.cgi?id=1311429 (no_proxy variable ignored when configured in virt-who config file)

Version-Release number of selected component (if applicable):
# rpm -qa | egrep "virt-who|python-rhsm|python-rhsm-certificates|subscription-manager"
~~~
subscription-manager-plugin-container-1.19.21-1.el7.x86_64
subscription-manager-1.19.21-1.el7.x86_64
python-rhsm-1.19.9-1.el7.x86_64
virt-who-0.19-6.el7_4.noarch
python-rhsm-certificates-1.19.9-1.el7.x86_64
~~~

How reproducible:
100%

Steps to Reproduce:
1.  Proxy configured in /etc/rhsm/rhsm.conf

# grep proxy /etc/rhsm/rhsm.conf | grep -v "#" | grep -v "^$"
~~~
proxy_hostname = ktordeur-proxy.usersys.redhat.com
proxy_port = 3128
proxy_user = proxy
proxy_password = proxy
~~~

2. add NO_PROXY=* to virt-who config
# grep -i proxy /etc/sysconfig/virt-who /etc/virt-who.conf  | grep -v "#" | grep -v "^$"
~~~
/etc/sysconfig/virt-who:NO_PROXY=*
/etc/virt-who.conf:NO_PROXY=*
~~~

3. Run virt-who
# virt-who -o -d


NOTE (workaround): It seems that when it's configured in rhsm.conf virt-who respects the no_proxy correctly if this is the case we should change our documentation to reflect the same.

# grep proxy /etc/rhsm/rhsm.conf | grep -v "#" | grep -v "^$"
~~~
proxy_hostname = ktordeur-proxy.usersys.redhat.com
proxy_port = 3128
proxy_user = proxy
proxy_password = proxy
no_proxy = provisioning.sysmgmt.lan
~~~

# virt-who -o -d
~~~
2017-08-29 12:21:29,279 [virtwho.init DEBUG] MainProcess(18109):MainThread @executor.py:__init__:52 - Using config named 'rhevm4'
2017-08-29 12:21:29,279 [virtwho.init INFO] MainProcess(18109):MainThread @main.py:main:183 - Using configuration "rhevm4" ("rhevm" mode)
2017-08-29 12:21:29,280 [virtwho.init INFO] MainProcess(18109):MainThread @main.py:main:185 - Using reporter_id=''
2017-08-29 12:21:29,320 [rhsm.https DEBUG] MainProcess(18109):MainThread @https.py:<module>:54 - Using standard libs to provide httplib and ssl
2017-08-29 12:21:29,327 [virtwho.rhevm4 DEBUG] MainProcess(18109):Thread-2 @virt.py:run:375 - Thread 'rhevm4' started
2017-08-29 12:21:29,831 [virtwho.rhevm4 INFO] MainProcess(18109):Thread-2 @virt.py:_send_data:888 - Report for config "rhevm4" gathered, placing in datastore
2017-08-29 12:21:29,831 [virtwho.rhevm4 DEBUG] MainProcess(18109):Thread-2 @virt.py:run:394 - Thread 'rhevm4' terminated
2017-08-29 12:21:30,330 [virtwho.destination_-7452366713885993190 DEBUG] MainProcess(18109):Thread-3 @virt.py:run:375 - Thread 'destination_-7452366713885993190' started
2017-08-29 12:21:30,332 [rhsm.connection DEBUG] MainProcess(18109):Thread-3 @connection.py:_fix_no_proxy:251 - Environment variable NO_PROXY=provisioning.sysmgmt.lan will be used
2017-08-29 12:21:30,332 [rhsm.connection INFO] MainProcess(18109):Thread-3 @connection.py:__init__:822 - Connection built: host=provisioning.sysmgmt.lan port=443 handler=/rhsm auth=basic username=virtwho
2017-08-29 12:21:30,331 [virtwho.destination_-7452366713885993190 DEBUG] MainProcess(18109):Thread-3 @subscriptionmanager.py:_connect:127 - Authenticating with RHSM username virtwho
2017-08-29 12:21:30,333 [rhsm.connection DEBUG] MainProcess(18109):Thread-3 @connection.py:_request:516 - Making request: GET /rhsm/status/
2017-08-29 12:21:30,432 [rhsm.connection INFO] MainProcess(18109):Thread-3 @connection.py:_request:552 - Response: status=200, request="GET /rhsm/status/"
2017-08-29 12:21:30,433 [rhsm.connection DEBUG] MainProcess(18109):Thread-3 @connection.py:_request:516 - Making request: GET /rhsm/status
2017-08-29 12:21:30,432 [virtwho.destination_-7452366713885993190 DEBUG] MainProcess(18109):Thread-3 @subscriptionmanager.py:hypervisorCheckIn:179 - Checking if server has capability 'hypervisor_async'
2017-08-29 12:21:30,528 [rhsm.connection INFO] MainProcess(18109):Thread-3 @connection.py:_request:552 - Response: status=200, request="GET /rhsm/status"
2017-08-29 12:21:30,528 [rhsm.connection DEBUG] MainProcess(18109):Thread-3 @connection.py:_load_manager_capabilities:867 - Server has the following capabilities: [u'cores', u'ram', u'instance_multiplier', u'derived_product', u'cert_v3', u'guest_limit', u'vcpu', u'storage_band']
2017-08-29 12:21:30,529 [virtwho.destination_-7452366713885993190 DEBUG] MainProcess(18109):Thread-3 @subscriptionmanager.py:hypervisorCheckIn:191 - Server does not have 'hypervisors_async' capability
2017-08-29 12:21:30,530 [rhsm.connection DEBUG] MainProcess(18109):Thread-3 @connection.py:_request:516 - Making request: POST /rhsm/hypervisors?owner=Default_Organization&env=Library
2017-08-29 12:21:30,529 [virtwho.destination_-7452366713885993190 INFO] MainProcess(18109):Thread-3 @subscriptionmanager.py:hypervisorCheckIn:202 - Sending update in hosts-to-guests mapping for config "destination_-7452366713885993190": 1 hypervisors and 1 guests found
2017-08-29 12:21:30,529 [virtwho.destination_-7452366713885993190 DEBUG] MainProcess(18109):Thread-3 @subscriptionmanager.py:hypervisorCheckIn:203 - Host-to-guest mapping: {
    "ktordeur-rhev4-hypervisor-rhvh": [
        {
            "guestId": "399e7d3f-818f-47d3-9ccc-76b1367b516a", 
            "state": 1, 
            "attributes": {
                "active": 1, 
                "virtWhoType": "rhevm"
            }
        }
    ]
}
2017-08-29 12:21:31,773 [rhsm.connection INFO] MainProcess(18109):Thread-3 @connection.py:_request:552 - Response: status=200, request="POST /rhsm/hypervisors?owner=Default_Organization&env=Library"
2017-08-29 12:21:31,774 [virtwho.destination_-7452366713885993190 DEBUG] MainProcess(18109):Thread-3 @virt.py:_send_data:712 - At least one report for each connected source has been sent. Terminating.
2017-08-29 12:21:31,774 [virtwho.destination_-7452366713885993190 DEBUG] MainProcess(18109):Thread-3 @virt.py:run:394 - Thread 'destination_-7452366713885993190' terminated
2017-08-29 12:21:32,334 [virtwho.main DEBUG] MainProcess(18109):MainThread @__main__.py:main:23 - virt-who terminated
2017-08-29 12:21:32,334 [virtwho.main DEBUG] MainProcess(18109):MainThread @executor.py:terminate:226 - virt-who is shutting down
~~~


Actual results:
# virt-who -o -d
2017-08-29 12:05:37,371 [virtwho.init DEBUG] MainProcess(7434):MainThread @executor.py:__init__:52 - Using config named 'rhevm4'
2017-08-29 12:05:37,372 [virtwho.init INFO] MainProcess(7434):MainThread @main.py:main:183 - Using configuration "rhevm4" ("rhevm" mode)
2017-08-29 12:05:37,372 [virtwho.init INFO] MainProcess(7434):MainThread @main.py:main:185 - Using reporter_id=''
2017-08-29 12:05:37,418 [rhsm.https DEBUG] MainProcess(7434):MainThread @https.py:<module>:54 - Using standard libs to provide httplib and ssl
2017-08-29 12:05:37,425 [virtwho.rhevm4 DEBUG] MainProcess(7434):Thread-2 @virt.py:run:375 - Thread 'rhevm4' started
2017-08-29 12:05:41,250 [virtwho.rhevm4 INFO] MainProcess(7434):Thread-2 @virt.py:_send_data:888 - Report for config "rhevm4" gathered, placing in datastore
2017-08-29 12:05:41,250 [virtwho.rhevm4 DEBUG] MainProcess(7434):Thread-2 @virt.py:run:394 - Thread 'rhevm4' terminated
2017-08-29 12:05:41,431 [virtwho.destination_-7452366713885993190 DEBUG] MainProcess(7434):Thread-3 @virt.py:run:375 - Thread 'destination_-7452366713885993190' started
2017-08-29 12:05:41,432 [rhsm.connection INFO] MainProcess(7434):Thread-3 @connection.py:__init__:822 - Connection built: http_proxy=ktordeur-proxy.usersys.redhat.com:3128 host=provisioning.sysmgmt.lan port=443 handler=/rhsm auth=basic username=virtwho
2017-08-29 12:05:41,434 [rhsm.connection DEBUG] MainProcess(7434):Thread-3 @connection.py:_request:501 - Using proxy: ktordeur-proxy.usersys.redhat.com:3128
2017-08-29 12:05:41,431 [virtwho.destination_-7452366713885993190 DEBUG] MainProcess(7434):Thread-3 @subscriptionmanager.py:_connect:127 - Authenticating with RHSM username virtwho
2017-08-29 12:05:41,434 [rhsm.connection DEBUG] MainProcess(7434):Thread-3 @connection.py:_request:516 - Making request: GET /rhsm/status/
2017-08-29 12:05:41,906 [virtwho.destination_-7452366713885993190 ERROR] MainProcess(7434):Thread-3 @virt.py:run:389 - Thread 'destination_-7452366713885993190' fails with exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/virtwho/virt/virt.py", line 380, in run
    self._run()
  File "/usr/lib/python2.7/site-packages/virtwho/virt/virt.py", line 336, in _run
    self._send_data(data_to_send)
  File "/usr/lib/python2.7/site-packages/virtwho/virt/virt.py", line 609, in _send_data
    options=self.options)
  File "/usr/lib/python2.7/site-packages/virtwho/manager/subscriptionmanager/subscriptionmanager.py", line 178, in hypervisorCheckIn
    self._connect(report.config)
  File "/usr/lib/python2.7/site-packages/virtwho/manager/subscriptionmanager/subscriptionmanager.py", line 140, in _connect
    if not self.connection.ping()['result']:
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 883, in ping
    return self.conn.request_get("/status/")
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 646, in request_get
    return self._request("GET", method, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 672, in _request
    info=info, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 528, in _request
    conn.request(request_type, handler, body=body, headers=final_headers)
  File "/usr/lib64/python2.7/httplib.py", line 1017, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1051, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1013, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 864, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 826, in send
    self.connect()
  File "/usr/lib64/python2.7/httplib.py", line 1227, in connect
    HTTPConnection.connect(self)
  File "/usr/lib64/python2.7/httplib.py", line 810, in connect
    self._tunnel()
  File "/usr/lib64/python2.7/httplib.py", line 792, in _tunnel
    message.strip()))
error: Tunnel connection failed: 503 Service Unavailable
2017-08-29 12:05:41,972 [virtwho.destination_-7452366713885993190 INFO] MainProcess(7434):Thread-3 @virt.py:_send_data:563 - Error report received
2017-08-29 12:05:41,972 [virtwho.destination_-7452366713885993190 DEBUG] MainProcess(7434):Thread-3 @virt.py:run:403 - Thread 'destination_-7452366713885993190' stopped after running once
2017-08-29 12:05:42,433 [virtwho.main DEBUG] MainProcess(7434):MainThread @__main__.py:main:23 - virt-who terminated
2017-08-29 12:05:42,433 [virtwho.main DEBUG] MainProcess(7434):MainThread @executor.py:terminate:226 - virt-who is shutting down


Expected results:
virt-who not to use the proxy when NO_PROXY=* is configured in virt-who config.

Additional info:
Comment 3 Eko 2017-08-30 07:23:34 UTC 
There is a comment from kevin why NO_PROXY is not added to /etc/virt-who/sysconfig:

https://bugzilla.redhat.com/show_bug.cgi?id=1299643#c30
Comment 4 Chris Snyder 2017-08-31 14:02:46 UTC 
NO_PROXY is not a key read by virt-who in /etc/virt-who.conf.
The NO_PROXY environment variable is effective for virt-who in /etc/sysconfig/virt-who when virt-who is run as a service (as the env vars in /etc/sysconfig/virt-who are included in the environment then).


Does this still break for you if you set the env var NO_PROXY=* in the terminal from which you run virt-who as well?

`NO_PROXY=* virt-who -o -d`

OR

when virt-who is run as a service?
Comment 5 Kenny Tordeurs 2017-08-31 14:16:28 UTC 
Hi Chris, 

When I run virt-who as a service it works as expected.
Comment 6 Chris Snyder 2017-08-31 14:47:41 UTC 
Kenny,

In that case we have two options that I can provide:

1) We can close this bug as NOTABUG

2) We can convert this bug to an RFE to read the no_proxy setting in /etc/virt-who.conf (thus enabling the behaviour desired).


Cheers,
Chris
Comment 7 Kenny Tordeurs 2017-09-01 06:50:42 UTC 
Chris,

I'd propose we convert it into an RFE as I know lots of customers are using `virt-who -o`

Thanks
Comment 10 William Poteat 2019-02-21 14:26:44 UTC 
The entry in the /etc/virt-who.conf file will be defined as a default variable.
It can also be applied to individual conf files and the setting can be different for each.
The name is 'rhsm_no_proxy' as the proxy is used in the subscription manager communication with the entitlement server and cdn.
Comment 14 errata-xmlrpc 2019-08-06 12:40:46 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2070