Bug 2056896

Summary: subscription-manager prints traceback to rhsm.log during registration, when VM is running on Hyper-V
Product: Red Hat Enterprise Linux 8 Reporter: Jiri Hnidek <jhnidek>
Component: subscription-managerAssignee: Jiri Hnidek <jhnidek>
Status: CLOSED ERRATA QA Contact: Red Hat subscription-manager QE Team <rhsm-qe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.6CC: jsefler, jstavel, rcavalca, redakkan
Target Milestone: rcKeywords: Triaged
Target Release: 8.7   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: subscription-manager-1.28.30-1.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2078827 (view as bug list) Environment:
Last Closed: 2022-11-08 10:48:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2078827    

Description Jiri Hnidek 2022-02-22 10:16:52 UTC
Description of problem:
When system is running on virtual machine and Hyper-V is used as hypervisor, then subscription-manager can print traceback to rhsm.log. We can see this issue, when there is machine in LAN with following address 169.254.169.254 and it ignores HTTP requests. This happens, because subscription-manager tries to detect public cloud using heuristics from system facts. When Hyper-V is used as hypervisor, then it false positive detection can happen. Thus subscription-manager tries to gather metadata from IMDS server.


Version-Release number of selected component (if applicable):

[root@localhost ~]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 3.2.22-1
subscription management rules: 5.41
subscription-manager: 1.28.21-3.el8


How reproducible:
100%


Steps to Reproduce:
1. Try to register system


Actual results:
System is registered, but following traceback is printed to rhsm.log


Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/subscription_manager/managercli.py", line 1863, in _do_command
    consumer = service.register(
  File "/usr/lib/python3/dist-packages/rhsmlib/services/register.py", line 85, in register
    facts_dict = self.facts.get_facts()
  File "/usr/lib/python3/dist-packages/subscription_manager/facts.py", line 78, in get_facts
    facts = collector.get_all()
  File "/usr/lib/python3/dist-packages/rhsmlib/facts/all.py", line 41, in get_all
    fact_collector = fact_collector_cls(collected_hw_info=results)
  File "/usr/lib/python3/dist-packages/rhsmlib/facts/cloud_facts.py", line 46, in __init__
    self.cloud_provider = get_cloud_provider(self._collected_hw_info)
  File "/usr/lib/python3/dist-packages/cloud_what/provider.py", line 149, in get_cloud_provider
    metadata = cloud_provider.get_metadata()
  File "/usr/lib/python3/dist-packages/cloud_what/providers/azure.py", line 235, in get_metadata
    return super(AzureCloudProvider, self).get_metadata()
  File "/usr/lib/python3/dist-packages/cloud_what/_base_provider.py", line 474, in get_metadata
    return self._get_metadata_from_server()
  File "/usr/lib/python3/dist-packages/cloud_what/providers/azure.py", line 195, in _get_metadata_from_server
    metadata = super(AzureCloudProvider, self)._get_metadata_from_server()
  File "/usr/lib/python3/dist-packages/cloud_what/_base_provider.py", line 402, in _get_metadata_from_server
    self._cached_metadata = self._get_data_from_server(
  File "/usr/lib/python3/dist-packages/cloud_what/providers/azure.py", line 188, in _get_data_from_server
    return super(AzureCloudProvider, self)._get_data_from_server(data_type, url, headers)
  File "/usr/lib/python3/dist-packages/cloud_what/_base_provider.py", line 387, in _get_data_from_server
    response = self._session.send(prepared_http_req, timeout=self.TIMEOUT)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='169.254.169.254', port=80): Read timed out. (read timeout=10.0)


Expected results:
No traceback is printed to rhsm.log


Additional info:
System facts contains following informations:

{
  ...
  "dmi.chassis.manufacturer": "microsoft",
  "virt.host_type": "hyperv",
  "virt.is_guest": true,
  ...
}

This BZ was created based on the discussion in this community PR: https://github.com/candlepin/subscription-manager/pull/2947

Comment 1 Jan Stavel 2022-05-11 16:34:30 UTC
I have verified that traceback is handled properly.

subscription management server: 3.2.22-1
subscription management rules: 5.41
subscription-manager: 1.28.29+28.g9ad67749b-1.git.0.5c4e029

I see: 

2022-05-12 00:16:05,887 [DEBUG] subscription-manager:152521:MainThread @custom.py:87 - Loading custom facts from: /etc/rhsm/facts/override.facts
2022-05-12 00:16:05,887 [DEBUG] subscription-manager:152521:MainThread @provider.py:104 - Trying to detect cloud provider
2022-05-12 00:16:05,888 [DEBUG] subscription-manager:152521:MainThread @provider.py:122 - No cloud provider detected using strong signs
2022-05-12 00:16:05,888 [DEBUG] subscription-manager:152521:MainThread @provider.py:140 - Cloud provider aws has probability: 0.0
2022-05-12 00:16:05,888 [DEBUG] subscription-manager:152521:MainThread @provider.py:140 - Cloud provider azure has probability: 0.7
2022-05-12 00:16:05,889 [DEBUG] subscription-manager:152521:MainThread @provider.py:140 - Cloud provider gcp has probability: 0.0
2022-05-12 00:16:05,889 [DEBUG] subscription-manager:152521:MainThread @provider.py:152 - Following cloud providers detected using heuristics: azure
2022-05-12 00:16:05,889 [DEBUG] subscription-manager:152521:MainThread @_base_provider.py:372 - Trying to get metadata from http://10.8.29.160/metadata/instance?api-version=2021-02-01
2022-05-12 00:16:15,906 [DEBUG] subscription-manager:152521:MainThread @_base_provider.py:386 - Unable to get azure metadata: HTTPConnectionPool(host='10.8.29.160', port=80): Max retries exceeded with url: /metadata/instance?api-version=2021-02-01 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f079d3bd278>, 'Connection to 10.8.29.160 timed out. (connect timeout=10.0)'))
2022-05-12 00:16:15,910 [DEBUG] subscription-manager:152521:MainThread @_base_provider.py:372 - Trying to get api_versions from http://10.8.29.160/metadata/versions
2022-05-12 00:16:25,957 [DEBUG] subscription-manager:152521:MainThread @_base_provider.py:386 - Unable to get azure api_versions: HTTPConnectionPool(host='10.8.29.160', port=80): Max retries exceeded with url: /metadata/versions (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f079d350048>, 'Connection to 10.8.29.160 timed out. (connect timeout=10.0)'))
2022-05-12 00:16:25,959 [ERROR] subscription-manager:152521:MainThread @azure.py:153 - Unable to decode Azure API versions: the JSON object must be str, bytes or bytearray, not 'NoneType'
2022-05-12 00:16:25,960 [DEBUG] subscription-manager:152521:MainThread @provider.py:194 - Unable to get metadata from any cloud provider detected using heuristics
2022-05-12 00:16:25,960 [DEBUG] subscription-manager:152521:MainThread @profile.py:100 - This system is not running on AWS. Skipping fixing of RHUI repos.


override.facts:

          {
          "dmi.chassis.foo": "microsoft",
          "virt.host_type": "hyperv",
          "virt.is_guest": true,
          "dmi.chassis.bar": "azure"
          }


I set a connection IP address in a file /usr/lib64/python3.6/site-packages/cloud_what/providers/azure.py.

   # ORIGIN BASE_CLOUD_PROVIDER_METADATA_URL = "http://169.254.169.254/metadata/instance?api-version="
   BASE_CLOUD_PROVIDER_METADATA_URL = "http://10.8.29.160/metadata/instance?api-version="

for all occurrences of the original IP.

the new IP points to a service that drops packets incomming to port 80 silently.

Comment 5 Jan Stavel 2022-05-24 16:49:34 UTC
verified.  
I have played the scenario described above.

ie:  
- use override.facts for subscription-manager to think the system runs on Azure
- register the system

 
I see in /var/log/rhsm/rhsm.log:

2022-05-24 22:16:55,664 [DEBUG] rhsmcertd-worker:98739:MainThread @custom.py:87 - Loading custom facts from: /etc/rhsm/facts/override.facts
2022-05-24 22:16:55,665 [DEBUG] rhsmcertd-worker:98739:MainThread @provider.py:104 - Trying to detect cloud provider
2022-05-24 22:16:55,665 [DEBUG] rhsmcertd-worker:98739:MainThread @provider.py:122 - No cloud provider detected using strong signs
2022-05-24 22:16:55,666 [DEBUG] rhsmcertd-worker:98739:MainThread @provider.py:140 - Cloud provider aws has probability: 0.0
2022-05-24 22:16:55,666 [DEBUG] rhsmcertd-worker:98739:MainThread @provider.py:140 - Cloud provider azure has probability: 0.7
2022-05-24 22:16:55,667 [DEBUG] rhsmcertd-worker:98739:MainThread @provider.py:140 - Cloud provider gcp has probability: 0.0
2022-05-24 22:16:55,667 [DEBUG] rhsmcertd-worker:98739:MainThread @provider.py:152 - Following cloud providers detected using heuristics: azure
2022-05-24 22:16:55,667 [DEBUG] rhsmcertd-worker:98739:MainThread @_base_provider.py:372 - Trying to get metadata from http://169.254.169.254/metadata/instance?api-version=2021-02-01
2022-05-24 22:16:55,710 [DEBUG] rhsmcertd-worker:98739:MainThread @_base_provider.py:386 - Unable to get azure metadata: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /metadata/instance?api-version=20
21-02-01 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2e1d997550>: Failed to establish a new connection: [Errno 113] No route to host',))

no traceback appears.

Comment 7 errata-xmlrpc 2022-11-08 10:48:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (subscription-manager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:7719