RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1922151 - It is not possible to gather AWS instance_id, when only IMDSv2 is enabled on AWS instance
Summary: It is not possible to gather AWS instance_id, when only IMDSv2 is enabled on ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: subscription-manager
Version: 8.3
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: 8.5
Assignee: Jiri Hnidek
QA Contact: Red Hat subscription-manager QE Team
URL:
Whiteboard:
Depends On: 1994718
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-29 11:16 UTC by Jiri Hnidek
Modified: 2021-11-10 07:52 UTC (History)
5 users (show)

Fixed In Version: subscription-manager-1.28.21-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-09 19:37:35 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Screenshot of AWS wizard (42.27 KB, image/png)
2021-01-29 11:20 UTC, Jiri Hnidek 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Github candlepin subscription-manager pull 2367 0 None closed Detecting of cloud providers; ENT-3288 2021-07-15 14:14:48 UTC
Github candlepin subscription-manager pull 2515 0 None closed Refactoring of cloud collector/detector and facts 2021-07-15 13:48:55 UTC
Github candlepin subscription-manager pull 2730 0 None None None 2021-08-11 10:53:24 UTC
Github candlepin subscription-manager pull 2736 0 None None None 2021-08-18 15:01:20 UTC
Github fedora-selinux selinux-policy pull 836 0 None None None 2021-08-18 15:01:20 UTC
Red Hat Product Errata RHBA-2021:4390 0 None None None 2021-11-09 19:37:50 UTC

Description Jiri Hnidek 2021-01-29 11:16:20 UTC 
Description of problem:
When AWS instance is created in AWS wizard, then it is possible to enabled only usage IMDSv2. When AWS VM tries to gather metadata, then it is not possible

Version-Release number of selected component (if applicable):

[root@ip-1-2-3-4 ~]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 3.2.11-1
subscription management rules: 5.41
subscription-manager: 1.29.5-1.git.7.336879d.el8

How reproducible: 100%


Steps to Reproduce:
1. Create AWS instance and enable only IMDSv2 in the AWS wizard.

2. Register system:

[root@ip-1-2-3-4 ~]# subscription-manager register --username <username> --password <secret_password>

3. Try to get system facts:

[root@ip-1-2-3-4 ~]#  subscription-manager facts | grep aws_instance_id

Actual results:


Expected results:
aws_instance_id: <i-yourinstanceid>

Additional info:

2021-01-29 11:09:03,159 [ERROR] subscription-manager:208699:MainThread @cloud_facts.py:81 - Cannot retrieve AWS instance Id: ('Failed to parse json data with error: %s', 'Expecting value: line 1 column 1 (char 0)')
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/rhsmlib/facts/cloud_facts.py", line 107, in parse_content
    return json.loads(content)
  File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/rhsmlib/facts/cloud_facts.py", line 78, in get_aws_instance_id
    values = self.parse_content(output)
  File "/usr/lib64/python3.6/site-packages/rhsmlib/facts/cloud_facts.py", line 109, in parse_content
    raise ValueError('Failed to parse json data with error: %s', str(e))
ValueError: ('Failed to parse json data with error: %s', 'Expecting value: line 1 column 1 (char 0)')
2021-01-29 11:09:03,160 [DEBUG] subscription-manager:208699:MainThread @repolib.py:172 - The rhsm.auto_enable_yum_plugins is disabled. Skipping the enablement of yum plugins.
Comment 1 Jiri Hnidek 2021-01-29 11:20:42 UTC 
Created attachment 1752005 [details]
Screenshot of AWS wizard
Comment 8 Jiri Hnidek 2021-08-11 10:53:24 UTC 
Issues with two many calls to IMDS server is solved in this PR: https://github.com/candlepin/subscription-manager/pull/2730
Comment 16 Craig Donnelly 2021-08-27 00:55:05 UTC 
>> Testing for final verification with latest errata build: subscription-manager-1.21.28-2

>> Still have selinux error, and it does prevent auto-registration:

2021-08-27 00:26:56,611 [ERROR] rhsmcertd-worker:13796:MainThread @rhsmcertd_worker.py:289 - [Errno 13] Permission denied: '/var/cache/cloud-what/aws_token.json'
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/subscription_manager/scripts/rhsmcertd_worker.py", line 278, in main
    _main(options, log)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/scripts/rhsmcertd_worker.py", line 203, in _main
    _auto_register(cp_provider, log)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/scripts/rhsmcertd_worker.py", line 147, in _auto_register
    cloud_info = _collect_cloud_info(cloud_list, log)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/scripts/rhsmcertd_worker.py", line 87, in _collect_cloud_info
    metadata: Union[str, None] = cloud_provider.get_metadata()
  File "/usr/lib64/python3.6/site-packages/cloud_what/providers/aws.py", line 385, in get_metadata
    return super(AWSCloudProvider, self).get_metadata()
  File "/usr/lib64/python3.6/site-packages/cloud_what/_base_provider.py", line 470, in get_metadata
    return self._get_metadata_from_server()
  File "/usr/lib64/python3.6/site-packages/cloud_what/providers/aws.py", line 306, in _get_metadata_from_server
    return self._get_metadata_from_server_imds_v2()
  File "/usr/lib64/python3.6/site-packages/cloud_what/providers/aws.py", line 267, in _get_metadata_from_server_imds_v2
    token = self._get_token()
  File "/usr/lib64/python3.6/site-packages/cloud_what/providers/aws.py", line 242, in _get_token
    token = self._get_token_from_server()
  File "/usr/lib64/python3.6/site-packages/cloud_what/providers/aws.py", line 214, in _get_token_from_server
    self._write_token_to_cache_file()
  File "/usr/lib64/python3.6/site-packages/cloud_what/_base_provider.py", line 205, in _write_token_to_cache_file
    with open(self.TOKEN_CACHE_FILE, "w") as token_cache_file:
PermissionError: [Errno 13] Permission denied: '/var/cache/cloud-what/aws_token.json'

>> We will need to add a release note for a workaround to this issue.

>> Completing testing using a short workaround mentioned in comment #12.

>> Collected IMDSv2:
2021-08-27 00:45:24,335 [DEBUG] rhsmcertd-worker:13901:MainThread @provider.py:97 - Detected one cloud provider using strong signs: aws
2021-08-27 00:45:24,335 [DEBUG] rhsmcertd-worker:13901:MainThread @aws.py:250 - Trying to get AWS metadata from http://169.254.169.254/latest/dynamic/instance-identity/document using IMDSv1
2021-08-27 00:45:24,335 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:372 - Trying to get metadata from http://169.254.169.254/latest/dynamic/instance-identity/document
2021-08-27 00:45:24,340 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:391 - Unable to get aws metadata: 401
2021-08-27 00:45:24,340 [DEBUG] rhsmcertd-worker:13901:MainThread @aws.py:265 - Trying to get AWS metadata from http://169.254.169.254/latest/dynamic/instance-identity/document using IMDSv2
2021-08-27 00:45:24,340 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:253 - Reading cache file with aws token: /var/cache/cloud-what/aws_token.json
2021-08-27 00:45:24,340 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:256 - Cache file: /var/cache/cloud-what/aws_token.json with aws token does not exist
2021-08-27 00:45:24,340 [DEBUG] rhsmcertd-worker:13901:MainThread @aws.py:193 - Requesting AWS token from http://169.254.169.254/latest/api/token
2021-08-27 00:45:24,343 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:203 - Writing aws token to file /var/cache/cloud-what/aws_token.json
2021-08-27 00:45:24,343 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:372 - Trying to get metadata from http://169.254.169.254/latest/dynamic/instance-identity/document
2021-08-27 00:45:24,345 [DEBUG] rhsmcertd-worker:13901:MainThread @aws.py:332 - Trying to get AWS signature from http://169.254.169.254/latest/dynamic/instance-identity/rsa2048 using IMDSv2
2021-08-27 00:45:24,346 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:372 - Trying to get signature from http://169.254.169.254/latest/dynamic/instance-identity/rsa2048
2021-08-27 00:45:24,348 [INFO] rhsmcertd-worker:13901:MainThread @rhsmcertd_worker.py:104 - Metadata and signature gathered for cloud provider: aws
<SNIP>
2021-08-27 00:45:24,863 [DEBUG] rhsmcertd-worker:13901:MainThread @_base_provider.py:463 - Using metadata from in-memory cache
2021-08-27 00:45:24,863 [DEBUG] rhsmcertd-worker:13901:MainThread @connection.py:710 - Making request: POST /subscription/consumers
2021-08-27 00:45:24,864 [DEBUG] rhsmcertd-worker:13901:MainThread @connection.py:566 - Loaded CA certificates from /etc/rhsm/ca/: redhat-entitlement-authority.pem, redhat-uep.pem
2021-08-27 00:45:27,918 [DEBUG] rhsmcertd-worker:13901:MainThread @connection.py:806 - Response time: 0.2699291706085205, Smoothed response time: 0.27456166744232174
2021-08-27 00:45:27,919 [DEBUG] rhsmcertd-worker:13901:MainThread @connection.py:783 - Response: status=200, requestUuid=0e4dec6f-97cc-4abd-8a52-2e3927edf29d, request="POST /subscription/consumers"
2021-08-27 00:45:27,920 [DEBUG] rhsmcertd-worker:13901:MainThread @cache.py:120 - Wrote cache: /var/lib/rhsm/cache/installed_products.json
2021-08-27 00:45:27,921 [INFO] rhsmcertd-worker:13901:MainThread @managerlib.py:72 - Consumer created: ip-10-0-0-222.us-east-2.compute.internal (facefa97-7624-4c78-9097-224f07f67d92)
2021-08-27 00:45:27,921 [DEBUG] rhsmcertd-worker:13901:MainThread @identity.py:139 - Loading consumer info from identity certificates.
2021-08-27 00:45:27,922 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:351 - Successfully read local syspurpose contents.
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:388 - Successfully read cached syspurpose contents.
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:661 - Attempting a three-way merge...
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:693 - Three way merge: remote value was changed for key 'addons'.
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @syspurposelib.py:195 - local result: {} 
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:351 - Successfully read local syspurpose contents.
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:388 - Successfully read cached syspurpose contents.
2021-08-27 00:45:27,923 [DEBUG] rhsmcertd-worker:13901:MainThread @files.py:591 - Successfully updated syspurpose values at '/etc/rhsm/syspurpose/syspurpose.json'.
2021-08-27 00:45:27,924 [DEBUG] rhsmcertd-worker:13901:MainThread @rhsmcertd_worker.py:177 - Auto-registration performed successfully

>> Cache gets used, registration is successful for IMDSv2 only instances.

>> Need note for selinux workaround on 8.5 release, preferably using an audit .pp workaround, but the direct workaround works fine as well.
($ sudo chcon -t rhsmcertd_var_lib_t /var/cache/cloud-what/)

Verified.
Comment 18 errata-xmlrpc 2021-11-09 19:37:35 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (subscription-manager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4390
Note You need to log in before you can comment on or make changes to this bug.