Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1922151

Summary: It is not possible to gather AWS instance_id, when only IMDSv2 is enabled on AWS instance
Product: Red Hat Enterprise Linux 8 Reporter: Jiri Hnidek <jhnidek>
Component: subscription-managerAssignee: Jiri Hnidek <jhnidek>
Status: CLOSED ERRATA QA Contact: Red Hat subscription-manager QE Team <rhsm-qe>
Severity: medium Docs Contact:
Priority: medium    
Version: 8.3CC: cdonnell, csnyder, linl, redakkan, xiliang
Target Milestone: rcKeywords: Triaged
Target Release: 8.5Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: subscription-manager-1.28.21-1.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-09 19:37:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1994718    
Bug Blocks:    
Attachments:
Description Flags
Screenshot of AWS wizard none

Description Jiri Hnidek 2021-01-29 11:16:20 UTC 
Description of problem:
When AWS instance is created in AWS wizard, then it is possible to enabled only usage IMDSv2. When AWS VM tries to gather metadata, then it is not possible

Version-Release number of selected component (if applicable):

[root@ip-1-2-3-4 ~]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 3.2.11-1
subscription management rules: 5.41
subscription-manager: 1.29.5-1.git.7.336879d.el8

How reproducible: 100%


Steps to Reproduce:
1. Create AWS instance and enable only IMDSv2 in the AWS wizard.

2. Register system:

[root@ip-1-2-3-4 ~]# subscription-manager register --username <username> --password <secret_password>

3. Try to get system facts:

[root@ip-1-2-3-4 ~]#  subscription-manager facts | grep aws_instance_id

Actual results:


Expected results:
aws_instance_id: <i-yourinstanceid>

Additional info:

2021-01-29 11:09:03,159 [ERROR] subscription-manager:208699:MainThread @cloud_facts.py:81 - Cannot retrieve AWS instance Id: ('Failed to parse json data with error: %s', 'Expecting value: line 1 column 1 (char 0)')
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/rhsmlib/facts/cloud_facts.py", line 107, in parse_content
    return json.loads(content)
  File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/rhsmlib/facts/cloud_facts.py", line 78, in get_aws_instance_id
    values = self.parse_content(output)
  File "/usr/lib64/python3.6/site-packages/rhsmlib/facts/cloud_facts.py", line 109, in parse_content
    raise ValueError('Failed to parse json data with error: %s', str(e))
ValueError: ('Failed to parse json data with error: %s', 'Expecting value: line 1 column 1 (char 0)')
2021-01-29 11:09:03,160 [DEBUG] subscription-manager:208699:MainThread @repolib.py:172 - The rhsm.auto_enable_yum_plugins is disabled. Skipping the enablement of yum plugins.
Comment 1 Jiri Hnidek 2021-01-29 11:20:42 UTC 
Created attachment 1752005 [details]
Screenshot of AWS wizard
Comment 8 Jiri Hnidek 2021-08-11 10:53:24 UTC 
Issues with two many calls to IMDS server is solved in this PR: https://github.com/candlepin/subscription-manager/pull/2730
Comment 16 Craig Donnelly 2021-08-27 00:55:05 UTC 
>> Testing for final verification with latest errata build: subscription-manager-1.21.28-2

>> Still have selinux error, and it does prevent auto-registration:

2021-08-27 00:26:56,611 [ERROR] rhsmcertd-worker:13796:MainThread @rhsmcertd_worker.py:289 - [Errno 13] Permission denied: '/var/cache/cloud-what/aws_token.json'
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/subscription_manager/scripts/rhsmcertd_worker.py", line 278, in main
    _main(options, log)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/scripts/rhsmcertd_worker.py", line 203, in _main
    _auto_register(cp_provider, log)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/scripts/rhsmcertd_worker.py", line 147, in _auto_register
    cloud_info = _collect_cloud_info(cloud_list, log)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/scripts/rhsmcertd_worker.py", line 87, in _collect_cloud_info
    metadata: Union[str, None] = cloud_provider.get_metadata()
  File "/usr/lib64/python3.6/site-packages/cloud_what/providers/aws.py", line 385, in get_metadata
    return super(AWSCloudProvider, self).get_metadata()
  File "/usr/lib64/python3.6/site-packages/cloud_what/_base_provider.py", line 470, in get_metadata
    return self._get_metadata_from_server()
  File "/usr/lib64/python3.6/site-packages/cloud_what/providers/aws.py", line 306, in _get_metadata_from_server
    return self._get_metadata_from_server_imds_v2()
  File "/usr/lib64/python3.6/site-packages/cloud_what/providers/aws.py", line 267, in _get_metadata_from_server_imds_v2
    token = self._get_token()
  File "/usr/lib64/python3.6/site-packages/cloud_what/providers/aws.py", line 242, in _get_token
    token = self._get_token_from_server()
  File "/usr/lib64/python3.6/site-packages/cloud_what/providers/aws.py", line 214, in _get_token_from_server
    self._write_token_to_cache_file()
  File "/usr/lib64/python3.6/site-packages/cloud_what/_base_provider.py", line 205, in _write_token_to_cache_file
    with open(self.TOKEN_CACHE_FILE, "w") as token_cache_file:
PermissionError: [Errno 13] Permission denied: '/var/cache/cloud-what/aws_token.json'

>> We will need to add a release note for a workaround to this issue.

>> Completing testing using a short workaround mentioned in comment #12.

>> Collected IMDSv2:
2021-08-27 00:45:24,335 [DEBUG] rhsmcertd-worker:13901:MainThread @provider.py:97 - Detected one cloud provider using strong signs: aws
2021-08-27 00:45:24,335 [DEBUG] rhsmcertd-worker:13901:MainThread @aws.py:250 - Trying to get AWS metadata from http://169.254.169.254/latest/dynamic/instance-identity/document using IMDSv1
2021-08-27 00:45:24,335 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:372 - Trying to get metadata from http://169.254.169.254/latest/dynamic/instance-identity/document
2021-08-27 00:45:24,340 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:391 - Unable to get aws metadata: 401
2021-08-27 00:45:24,340 [DEBUG] rhsmcertd-worker:13901:MainThread @aws.py:265 - Trying to get AWS metadata from http://169.254.169.254/latest/dynamic/instance-identity/document using IMDSv2
2021-08-27 00:45:24,340 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:253 - Reading cache file with aws token: /var/cache/cloud-what/aws_token.json
2021-08-27 00:45:24,340 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:256 - Cache file: /var/cache/cloud-what/aws_token.json with aws token does not exist
2021-08-27 00:45:24,340 [DEBUG] rhsmcertd-worker:13901:MainThread @aws.py:193 - Requesting AWS token from http://169.254.169.254/latest/api/token
2021-08-27 00:45:24,343 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:203 - Writing aws token to file /var/cache/cloud-what/aws_token.json
2021-08-27 00:45:24,343 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:372 - Trying to get metadata from http://169.254.169.254/latest/dynamic/instance-identity/document
2021-08-27 00:45:24,345 [DEBUG] rhsmcertd-worker:13901:MainThread @aws.py:332 - Trying to get AWS signature from http://169.254.169.254/latest/dynamic/instance-identity/rsa2048 using IMDSv2
2021-08-27 00:45:24,346 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:372 - Trying to get signature from http://169.254.169.254/latest/dynamic/instance-identity/rsa2048
2021-08-27 00:45:24,348 [INFO] rhsmcertd-worker:13901:MainThread @rhsmcertd_worker.py:104 - Metadata and signature gathered for cloud provider: aws
<SNIP>
2021-08-27 00:45:24,863 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:463 - Using metadata from in-memory cache
2021-08-27 00:45:24,863 [DEBUG] rhsmcertd-worker:13901:MainThread @connection.py:710 - Making request: POST /subscription/consumers
2021-08-27 00:45:24,864 [DEBUG] rhsmcertd-worker:13901:MainThread @connection.py:566 - Loaded CA certificates from /etc/rhsm/ca/: redhat-entitlement-authority.pem, redhat-uep.pem
2021-08-27 00:45:27,918 [DEBUG] rhsmcertd-worker:13901:MainThread @connection.py:806 - Response time: 0.2699291706085205, Smoothed response time: 0.27456166744232174
2021-08-27 00:45:27,919 [DEBUG] rhsmcertd-worker:13901:MainThread @connection.py:783 - Response: status=200, requestUuid=0e4dec6f-97cc-4abd-8a52-2e3927edf29d, request="POST /subscription/consumers"
2021-08-27 00:45:27,920 [DEBUG] rhsmcertd-worker:13901:MainThread @cache.py:120 - Wrote cache: /var/lib/rhsm/cache/installed_products.json
2021-08-27 00:45:27,921 [INFO] rhsmcertd-worker:13901:MainThread @managerlib.py:72 - Consumer created: ip-10-0-0-222.us-east-2.compute.internal (facefa97-7624-4c78-9097-224f07f67d92)
2021-08-27 00:45:27,921 [DEBUG] rhsmcertd-worker:13901:MainThread @identity.py:139 - Loading consumer info from identity certificates.
2021-08-27 00:45:27,922 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:351 - Successfully read local syspurpose contents.
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:388 - Successfully read cached syspurpose contents.
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:661 - Attempting a three-way merge...
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:693 - Three way merge: remote value was changed for key 'addons'.
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @syspurposelib.py:195 - local result: {} 
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:351 - Successfully read local syspurpose contents.
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:388 - Successfully read cached syspurpose contents.
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:591 - Successfully updated syspurpose values at '/etc/rhsm/syspurpose/syspurpose.json'.
2021-08-27 00:45:27,924 [DEBUG] rhsmcertd-worker:13901:MainThread @rhsmcertd_worker.py:177 - Auto-registration performed successfully

>> Cache gets used, registration is successful for IMDSv2 only instances.

>> Need note for selinux workaround on 8.5 release, preferably using an audit .pp workaround, but the direct workaround works fine as well.
($ sudo chcon -t rhsmcertd_var_lib_t /var/cache/cloud-what/)

Verified.
Comment 18 errata-xmlrpc 2021-11-09 19:37:35 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (subscription-manager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4390