Bug 1922151

Summary: It is not possible to gather AWS instance_id, when only IMDSv2 is enabled on AWS instance
Product: Red Hat Enterprise Linux 8 Reporter: Jiri Hnidek <jhnidek>
Component: subscription-managerAssignee: Jiri Hnidek <jhnidek>
Status: CLOSED ERRATA QA Contact: Red Hat subscription-manager QE Team <rhsm-qe>
Severity: medium Docs Contact:
Priority: medium    
Version: 8.3CC: cdonnell, csnyder, linl, redakkan, xiliang
Target Milestone: rcKeywords: Triaged
Target Release: 8.5Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: subscription-manager-1.28.21-1.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-09 19:37:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1994718    
Bug Blocks:    
Attachments:
Description Flags
Screenshot of AWS wizard none

Description Jiri Hnidek 2021-01-29 11:16:20 UTC
Description of problem:
When AWS instance is created in AWS wizard, then it is possible to enabled only usage IMDSv2. When AWS VM tries to gather metadata, then it is not possible

Version-Release number of selected component (if applicable):

[root@ip-1-2-3-4 ~]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 3.2.11-1
subscription management rules: 5.41
subscription-manager: 1.29.5-1.git.7.336879d.el8

How reproducible: 100%


Steps to Reproduce:
1. Create AWS instance and enable only IMDSv2 in the AWS wizard.

2. Register system:

[root@ip-1-2-3-4 ~]# subscription-manager register --username <username> --password <secret_password>

3. Try to get system facts:

[root@ip-1-2-3-4 ~]#  subscription-manager facts | grep aws_instance_id

Actual results:


Expected results:
aws_instance_id: <i-yourinstanceid>

Additional info:

2021-01-29 11:09:03,159 [ERROR] subscription-manager:208699:MainThread @cloud_facts.py:81 - Cannot retrieve AWS instance Id: ('Failed to parse json data with error: %s', 'Expecting value: line 1 column 1 (char 0)')
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/rhsmlib/facts/cloud_facts.py", line 107, in parse_content
    return json.loads(content)
  File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/rhsmlib/facts/cloud_facts.py", line 78, in get_aws_instance_id
    values = self.parse_content(output)
  File "/usr/lib64/python3.6/site-packages/rhsmlib/facts/cloud_facts.py", line 109, in parse_content
    raise ValueError('Failed to parse json data with error: %s', str(e))
ValueError: ('Failed to parse json data with error: %s', 'Expecting value: line 1 column 1 (char 0)')
2021-01-29 11:09:03,160 [DEBUG] subscription-manager:208699:MainThread @repolib.py:172 - The rhsm.auto_enable_yum_plugins is disabled. Skipping the enablement of yum plugins.

Comment 1 Jiri Hnidek 2021-01-29 11:20:42 UTC
Created attachment 1752005 [details]
Screenshot of AWS wizard

Comment 8 Jiri Hnidek 2021-08-11 10:53:24 UTC
Issues with two many calls to IMDS server is solved in this PR: https://github.com/candlepin/subscription-manager/pull/2730

Comment 16 Craig Donnelly 2021-08-27 00:55:05 UTC
>> Testing for final verification with latest errata build: subscription-manager-1.21.28-2

>> Still have selinux error, and it does prevent auto-registration:

2021-08-27 00:26:56,611 [ERROR] rhsmcertd-worker:13796:MainThread @rhsmcertd_worker.py:289 - [Errno 13] Permission denied: '/var/cache/cloud-what/aws_token.json'
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/subscription_manager/scripts/rhsmcertd_worker.py", line 278, in main
    _main(options, log)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/scripts/rhsmcertd_worker.py", line 203, in _main
    _auto_register(cp_provider, log)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/scripts/rhsmcertd_worker.py", line 147, in _auto_register
    cloud_info = _collect_cloud_info(cloud_list, log)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/scripts/rhsmcertd_worker.py", line 87, in _collect_cloud_info
    metadata: Union[str, None] = cloud_provider.get_metadata()
  File "/usr/lib64/python3.6/site-packages/cloud_what/providers/aws.py", line 385, in get_metadata
    return super(AWSCloudProvider, self).get_metadata()
  File "/usr/lib64/python3.6/site-packages/cloud_what/_base_provider.py", line 470, in get_metadata
    return self._get_metadata_from_server()
  File "/usr/lib64/python3.6/site-packages/cloud_what/providers/aws.py", line 306, in _get_metadata_from_server
    return self._get_metadata_from_server_imds_v2()
  File "/usr/lib64/python3.6/site-packages/cloud_what/providers/aws.py", line 267, in _get_metadata_from_server_imds_v2
    token = self._get_token()
  File "/usr/lib64/python3.6/site-packages/cloud_what/providers/aws.py", line 242, in _get_token
    token = self._get_token_from_server()
  File "/usr/lib64/python3.6/site-packages/cloud_what/providers/aws.py", line 214, in _get_token_from_server
    self._write_token_to_cache_file()
  File "/usr/lib64/python3.6/site-packages/cloud_what/_base_provider.py", line 205, in _write_token_to_cache_file
    with open(self.TOKEN_CACHE_FILE, "w") as token_cache_file:
PermissionError: [Errno 13] Permission denied: '/var/cache/cloud-what/aws_token.json'

>> We will need to add a release note for a workaround to this issue.

>> Completing testing using a short workaround mentioned in comment #12.

>> Collected IMDSv2:
2021-08-27 00:45:24,335 [DEBUG] rhsmcertd-worker:13901:MainThread @provider.py:97 - Detected one cloud provider using strong signs: aws
2021-08-27 00:45:24,335 [DEBUG] rhsmcertd-worker:13901:MainThread @aws.py:250 - Trying to get AWS metadata from http://169.254.169.254/latest/dynamic/instance-identity/document using IMDSv1
2021-08-27 00:45:24,335 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:372 - Trying to get metadata from http://169.254.169.254/latest/dynamic/instance-identity/document
2021-08-27 00:45:24,340 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:391 - Unable to get aws metadata: 401
2021-08-27 00:45:24,340 [DEBUG] rhsmcertd-worker:13901:MainThread @aws.py:265 - Trying to get AWS metadata from http://169.254.169.254/latest/dynamic/instance-identity/document using IMDSv2
2021-08-27 00:45:24,340 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:253 - Reading cache file with aws token: /var/cache/cloud-what/aws_token.json
2021-08-27 00:45:24,340 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:256 - Cache file: /var/cache/cloud-what/aws_token.json with aws token does not exist
2021-08-27 00:45:24,340 [DEBUG] rhsmcertd-worker:13901:MainThread @aws.py:193 - Requesting AWS token from http://169.254.169.254/latest/api/token
2021-08-27 00:45:24,343 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:203 - Writing aws token to file /var/cache/cloud-what/aws_token.json
2021-08-27 00:45:24,343 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:372 - Trying to get metadata from http://169.254.169.254/latest/dynamic/instance-identity/document
2021-08-27 00:45:24,345 [DEBUG] rhsmcertd-worker:13901:MainThread @aws.py:332 - Trying to get AWS signature from http://169.254.169.254/latest/dynamic/instance-identity/rsa2048 using IMDSv2
2021-08-27 00:45:24,346 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:372 - Trying to get signature from http://169.254.169.254/latest/dynamic/instance-identity/rsa2048
2021-08-27 00:45:24,348 [INFO] rhsmcertd-worker:13901:MainThread @rhsmcertd_worker.py:104 - Metadata and signature gathered for cloud provider: aws
<SNIP>
2021-08-27 00:45:24,863 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:463 - Using metadata from in-memory cache
2021-08-27 00:45:24,863 [DEBUG] rhsmcertd-worker:13901:MainThread @connection.py:710 - Making request: POST /subscription/consumers
2021-08-27 00:45:24,864 [DEBUG] rhsmcertd-worker:13901:MainThread @connection.py:566 - Loaded CA certificates from /etc/rhsm/ca/: redhat-entitlement-authority.pem, redhat-uep.pem
2021-08-27 00:45:27,918 [DEBUG] rhsmcertd-worker:13901:MainThread @connection.py:806 - Response time: 0.2699291706085205, Smoothed response time: 0.27456166744232174
2021-08-27 00:45:27,919 [DEBUG] rhsmcertd-worker:13901:MainThread @connection.py:783 - Response: status=200, requestUuid=0e4dec6f-97cc-4abd-8a52-2e3927edf29d, request="POST /subscription/consumers"
2021-08-27 00:45:27,920 [DEBUG] rhsmcertd-worker:13901:MainThread @cache.py:120 - Wrote cache: /var/lib/rhsm/cache/installed_products.json
2021-08-27 00:45:27,921 [INFO] rhsmcertd-worker:13901:MainThread @managerlib.py:72 - Consumer created: ip-10-0-0-222.us-east-2.compute.internal (facefa97-7624-4c78-9097-224f07f67d92)
2021-08-27 00:45:27,921 [DEBUG] rhsmcertd-worker:13901:MainThread @identity.py:139 - Loading consumer info from identity certificates.
2021-08-27 00:45:27,922 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:351 - Successfully read local syspurpose contents.
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:388 - Successfully read cached syspurpose contents.
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:661 - Attempting a three-way merge...
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:693 - Three way merge: remote value was changed for key 'addons'.
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @syspurposelib.py:195 - local result: {} 
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:351 - Successfully read local syspurpose contents.
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:388 - Successfully read cached syspurpose contents.
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:591 - Successfully updated syspurpose values at '/etc/rhsm/syspurpose/syspurpose.json'.
2021-08-27 00:45:27,924 [DEBUG] rhsmcertd-worker:13901:MainThread @rhsmcertd_worker.py:177 - Auto-registration performed successfully

>> Cache gets used, registration is successful for IMDSv2 only instances.

>> Need note for selinux workaround on 8.5 release, preferably using an audit .pp workaround, but the direct workaround works fine as well.
($ sudo chcon -t rhsmcertd_var_lib_t /var/cache/cloud-what/)

Verified.

Comment 18 errata-xmlrpc 2021-11-09 19:37:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (subscription-manager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4390