Bug 2050751

Summary: fence_aws fails to find credentials when only IMDSv2 is enabled (RHEL7)
Product: Red Hat Enterprise Linux 7 Reporter: Oyvind Albrigtsen <oalbrigt>
Component: python-s3transferAssignee: Oyvind Albrigtsen <oalbrigt>
Status: CLOSED ERRATA QA Contact: Brandon Perkins <bperkins>
Severity: high Docs Contact:
Priority: high    
Version: 7.9CC: adamkam, bperkins, cfeist, devel, jered, jobaker, jreznik, nwahl, sbradley
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: 7.9Flags: pm-rhel: mirror+
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: python-s3transfer-0.1.13-1.el7_9.2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2048857
: 2052629 2052630 (view as bug list) Environment:
Last Closed: 2022-04-05 17:15:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2048857, 2050759, 2051935    
Bug Blocks: 2052629, 2052630    

Description Oyvind Albrigtsen 2022-02-04 15:08:37 UTC 
+++ This bug was initially created as a clone of Bug #2048857 +++

Description of problem:
The package "python*-botocore" should include "IMDSv2", but currently available package does not:

  Latest avaialble version:
    [root@clusterb-rhel8 python-botocore]# grep -i version $( find . | grep __init__.py )
    ./botocore-1.21.35/botocore/__init__.py:# Licensed under the Apache License, Version 2.0 (the "License"). You
    ./botocore-1.21.35/botocore/__init__.py:__version__ = '1.21.35'

IMDSv2 was introduced in Version 1.23.46:
     Add IMDS metadata token · boto/botocore@9297380 · GitHub
https://github.com/boto/botocore/commit/92973803db4a5389a915a75b21eb17b9096821a8#diff-d5dc8bb337890042943977b62e8bb2e7db71415d35bb17a41ad4bf4780fb3617

Several AWS specific resource agents and other packages ( such as "awscli" ) require python*-botocore:

    [root@clusterb-rhel8 ~]# repoquery -q --whatrequires python3-botocore
    awscli-0:1.14.50-5.el8.noarch
    python3-boto3-0:1.6.1-2.el8.noarch
    python3-s3transfer-0:0.1.13-1.el8.noarchpython3-boto3

        [root@clusterb-rhel8 ~]# repoquery -q --whatrequires python3-boto3
        fence-agents-aws-0:4.5.2-2.fc33.noarch
        fence-agents-aws-0:4.7.1-3.fc33.noarch
        lorax-composer-0:33.10-1.fc33.x86_64
        lorax-composer-0:33.11-1.fc33.x86_64
        nbdkit-S3-plugin-0:1.24.6-1.fc33.x86_64
        nodepool-0:3.12.0-2.fc33.noarch
        python3-certbot-dns-route53-0:1.19.0-1.fc33.noarch
        python3-certbot-dns-route53-0:1.7.0-1.fc33.noarch
        python3-django-storages+boto3-0:1.8-4.fc33.noarch
        python3-dns-lexicon+route53-0:3.3.17-5.fc33.noarch
        python3-rasterio+s3-0:1.2.10-1.fc33.x86_64
        python3-robosignatory-0:0.6.7-2.fc33.noarch
        python3-robosignatory-0:0.7.0-1.fc33.noarch
        python3-smart_open-0:5.1.0-1.fc33.noarch


        [root@clusterb-rhel8 ~]# repoquery -q --whatrequires python3-boto3
        fence-agents-aws-0:4.5.2-2.fc33.noarch
        fence-agents-aws-0:4.7.1-3.fc33.noarch
        lorax-composer-0:33.10-1.fc33.x86_64
        lorax-composer-0:33.11-1.fc33.x86_64
        nbdkit-S3-plugin-0:1.24.6-1.fc33.x86_64
        nodepool-0:3.12.0-2.fc33.noarch
        python3-certbot-dns-route53-0:1.19.0-1.fc33.noarch
        python3-certbot-dns-route53-0:1.7.0-1.fc33.noarch
        python3-django-storages+boto3-0:1.8-4.fc33.noarch
        python3-dns-lexicon+route53-0:3.3.17-5.fc33.noarch
        python3-rasterio+s3-0:1.2.10-1.fc33.x86_64
        python3-robosignatory-0:0.6.7-2.fc33.noarch
        python3-robosignatory-0:0.7.0-1.fc33.noarch
        python3-smart_open-0:5.1.0-1.fc33.noarch

        
    [root@clusterb-rhel8 ~]# repoquery -q --whatrequires python38-botocore

    [root@clusterb-rhel8 ~]# repoquery -q --whatrequires python2-botocore

    [root@clusterb-rhel8 ~]# repoquery -q --whatrequires python-botocore

These resource agents will fail intermittently without "IMDSv2" support. 


Version-Release number of selected component (if applicable):
python3-botocore-1.21.35-1
python38-botocore-1.21.35-2


How reproducible:


Steps to Reproduce:
1. Install "fence_aws" or awscli on Amazon AWS instance
2. Enable use of "IMDSv2"

   Use IMDSv2 - Amazon Elastic Compute Cloud
   https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html

3. Configure fencing agent to access instance via AIM role and using "IMDSv2".

4. Run a test fence.


Actual results:
Fencing agent is unable to find credentials using "IMDSv2"
~~~
# From fence_aws debug file:
2021-12-07 15:41:14,900 botocore.session DEBUG    Loading variable profile from defaults.
2021-12-07 15:41:14,900 botocore.session DEBUG    Loading variable config_file from defaults.
2021-12-07 15:41:14,903 botocore.session DEBUG    Loading variable credentials_file from defaults.
2021-12-07 15:41:14,903 botocore.session DEBUG    Loading variable data_path from defaults.
2021-12-07 15:41:14,907 botocore.loaders DEBUG    Loading JSON file: /usr/lib/python3.6/site-packages/boto3/data/ec2/2016-11-15/resources-1.json
2021-12-07 15:41:14,910 botocore.session DEBUG    Loading variable profile from defaults.
2021-12-07 15:41:14,910 botocore.session DEBUG    Loading variable ca_bundle from defaults.
2021-12-07 15:41:14,910 botocore.session DEBUG    Loading variable profile from defaults.
2021-12-07 15:41:14,910 botocore.session DEBUG    Loading variable credentials_file from defaults.
2021-12-07 15:41:14,910 botocore.session DEBUG    Loading variable config_file from defaults.
2021-12-07 15:41:14,910 botocore.session DEBUG    Loading variable profile from defaults.
2021-12-07 15:41:14,910 botocore.session DEBUG    Loading variable metadata_service_timeout from defaults.
2021-12-07 15:41:14,910 botocore.session DEBUG    Loading variable profile from defaults.
2021-12-07 15:41:14,910 botocore.session DEBUG    Loading variable metadata_service_num_attempts from defaults.
2021-12-07 15:41:14,914 botocore.session DEBUG    Loading variable profile from defaults.
2021-12-07 15:41:14,915 botocore.credentials DEBUG    Looking for credentials via: env
2021-12-07 15:41:14,915 botocore.credentials DEBUG    Looking for credentials via: assume-role
2021-12-07 15:41:14,915 botocore.credentials DEBUG    Looking for credentials via: shared-credentials-file
2021-12-07 15:41:14,915 botocore.credentials DEBUG    Looking for credentials via: custom-process
2021-12-07 15:41:14,915 botocore.credentials DEBUG    Looking for credentials via: config-file
2021-12-07 15:41:14,915 botocore.credentials DEBUG    Looking for credentials via: ec2-credentials-file
2021-12-07 15:41:14,915 botocore.credentials DEBUG    Looking for credentials via: boto-config
2021-12-07 15:41:14,916 botocore.credentials DEBUG    Looking for credentials via: container-role
2021-12-07 15:41:14,916 botocore.credentials DEBUG    Looking for credentials via: iam-role
2021-12-07 15:41:14,919 botocore.utils DEBUG    Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
~~~

Expected results:
Pulling instance credentials:
~~~
...
2019-09-19 04:23:38,570 DEBUG: Looking for credentials via: iam-role
2019-09-19 04:23:38,574 INFO: Starting new HTTP connection (1): 169.254.169.254
2019-09-19 04:23:38,575 DEBUG: "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 21
2019-09-19 04:23:38,576 INFO: Starting new HTTP connection (1): 169.254.169.254
2019-09-19 04:23:38,577 DEBUG: "GET /latest/meta-data/iam/security-credentials/power-user-role HTTP/1.1" 200 1298
2019-09-19 04:23:38,578 DEBUG: Found credentials from IAM Role: power-user-role
~~~
Additional info:
Amazon has reached on case 03136512 to request an update to botocore packages

--- Additional comment from Reid Wahl on 2022-02-01 10:39:34 CET ---

I think this is worth further inspection. When we added support for IMDSv2 in 2020-21, QE tested the list action for fence_aws, and it passed. See BZ1896827 and its clones. Also, the code that was added to fence_aws for IMDSv2 support applied only to get_instance_id(), which in turn is called only during a power-off or power-on action (not during the list action).

It's true that the patch wasn't backported to RHEL 8.1 and 8.2, but that's because the metadata call (where the token request is necessary) was added in RHEL 8.3. So the patch was irrelevant in 8.1/8.2. (It **was** backported to 7.6, 7.7, and 7.9.)

Presumably, QE wasn't using special boto* libraries (e.g., from upstream or pip). Rather, they used the libraries that get installed via yum/dnf as RPM packages.

It's possible that QE used locally configured credentials (e.g., ~/.aws/credentials) rather than an IAM role attached to the instance. They definitely didn't use explicit keys passed as fence_aws options.

If they used a credentials file and rather than an IAM role, then that may explain this behavior, since boto presumably needs to get a token to access the instance metadata to get the IAM role. In that case, it seems that we simply need to ship newer boto libs. (We can experiment of course; I have not yet done so.)

Otherwise (if we did test with a role, or if this issue happens even with other credential access methods), then this is rather mysterious.

-----

Reference on credential access methods:
  - https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html

As a note, we received a couple of end-customer support cases for this, until recently Adam from AWS opened a case requesting we provide updated boto libs for IMDSv2.

--- Additional comment from Reid Wahl on 2022-02-02 08:09:43 CET ---

This is indeed trivial to reproduce when relying on an instance IAM role for credentials (**without** other credentials stored on the VM somewhere like environment variables or an ~/.aws/credentials file). So it seems a newer boto is in order.

In my reproducer, the instance has an IAM role with a power user policy attached to it. The role has to be accessed via instance metadata since there are no other credentials present. Accessing instance metadata requires a token, which we don't have, if we're using IMDSv2 only.

# # Allow IMDSv1
[root@ip-10-0-0-243 ~]# aws ec2 modify-instance-metadata-options --instance-id i-062637239d3779fbc --http-tokens optional --http-endpoint enabled
{
...
}


# # Get the node list
[root@ip-10-0-0-243 ~]# fence_aws -r us-west-2 -o list --boto3_debug 1 -vvvvvv
2022-02-02 07:07:22,845 DEBUG: Boto debug level is 1 and sending debug info to /var/log/fence_aws_boto3.log
i-062637239d3779fbc,
i-0a5945b959a13b51e,

# # Check fence_aws_boto3.log
2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: env
2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: assume-role
2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: shared-credentials-file
2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: custom-process
2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: config-file
2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: ec2-credentials-file
2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: boto-config
2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: container-role
2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: iam-role
2022-02-02 07:07:22,871 botocore.credentials DEBUG    Found credentials from IAM Role: nwahl-ec2-rhel-ha


# # Disable IMDSv1
[root@ip-10-0-0-243 ~]# aws ec2 modify-instance-metadata-options --instance-id i-062637239d3779fbc --http-tokens required --http-endpoint enabled
{
...
}

# # Try to get the node list (it's empty)
[root@ip-10-0-0-243 ~]# fence_aws -r us-west-2 -o list --boto3_debug 1 -vvvvvv
2022-02-02 07:08:49,647 DEBUG: Boto debug level is 1 and sending debug info to /var/log/fence_aws_boto3.log
[root@ip-10-0-0-243 ~]# 

# # Check fence_aws_boto3.log
2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: env
2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: assume-role
2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: shared-credentials-file
2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: custom-process
2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: config-file
2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: ec2-credentials-file
2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: boto-config
2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: container-role
2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: iam-role
2022-02-02 07:08:49,669 botocore.utils DEBUG    Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
Comment 14 errata-xmlrpc 2022-04-05 17:15:20 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (python-s3transfer bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1195
Comment 15 Jason Woods 2022-04-07 12:49:51 UTC 
This update has triggered some severe failures on any code loading urllib3 before boto3.

# python
Python 2.7.5 (default, Nov 16 2020, 22:23:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib3
>>> import boto3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/boto3/__init__.py", line 16, in <module>
    from boto3.session import Session
  File "/usr/lib/python2.7/site-packages/boto3/session.py", line 19, in <module>
    import botocore.session
  File "/usr/lib/fence-agents/bundled/botocore/session.py", line 29, in <module>
    import botocore.credentials
  File "/usr/lib/fence-agents/bundled/botocore/credentials.py", line 34, in <module>
    from botocore.config import Config
  File "/usr/lib/fence-agents/bundled/botocore/config.py", line 16, in <module>
    from botocore.endpoint import DEFAULT_TIMEOUT, MAX_POOL_CONNECTIONS
  File "/usr/lib/fence-agents/bundled/botocore/endpoint.py", line 22, in <module>
    from botocore.awsrequest import create_request_object
  File "/usr/lib/fence-agents/bundled/botocore/awsrequest.py", line 24, in <module>
    import botocore.utils
  File "/usr/lib/fence-agents/bundled/botocore/utils.py", line 32, in <module>
    import botocore.httpsession
  File "/usr/lib/fence-agents/bundled/botocore/httpsession.py", line 10, in <module>
    from urllib3.util.ssl_ import (
ImportError: cannot import name PROTOCOL_TLS
>>>

Certbot from EPEL repository is currently broken after the package update to python-s3transfer due to the urllib3 update:

# certbot
An unexpected error occurred:
ImportError: cannot import name PROTOCOL_TLS
Please see the logfile '/tmp/tmpxyrTML/log' for more details.
# cat /tmp/tmpxyrTML/log
2022-04-07 12:10:52,605:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/bin/certbot", line 9, in <module>
    load_entry_point('certbot==1.11.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1383, in main
    plugins = plugins_disco.PluginsRegistry.find_all()
  File "/usr/lib/python2.7/site-packages/certbot/_internal/plugins/disco.py", line 236, in find_all
    plugin_ep = cls._load_entry_point(entry_point, plugins, with_prefix=False)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/plugins/disco.py", line 254, in _load_entry_point
    plugin_ep = PluginEntryPoint(entry_point, with_prefix)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/plugins/disco.py", line 56, in __init__
    self.plugin_cls = entry_point.load()
  File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/lib/python2.7/site-packages/certbot_dns_route53/authenticator.py", line 7, in <module>
    from certbot_dns_route53._internal import dns_route53
  File "/usr/lib/python2.7/site-packages/certbot_dns_route53/_internal/dns_route53.py", line 6, in <module>
    import boto3
  File "/usr/lib/python2.7/site-packages/boto3/__init__.py", line 16, in <module>
    from boto3.session import Session
  File "/usr/lib/python2.7/site-packages/boto3/session.py", line 19, in <module>
    import botocore.session
  File "/usr/lib/fence-agents/bundled/botocore/session.py", line 29, in <module>
    import botocore.credentials
  File "/usr/lib/fence-agents/bundled/botocore/credentials.py", line 34, in <module>
    from botocore.config import Config
  File "/usr/lib/fence-agents/bundled/botocore/config.py", line 16, in <module>
    from botocore.endpoint import DEFAULT_TIMEOUT, MAX_POOL_CONNECTIONS
  File "/usr/lib/fence-agents/bundled/botocore/endpoint.py", line 22, in <module>
    from botocore.awsrequest import create_request_object
  File "/usr/lib/fence-agents/bundled/botocore/awsrequest.py", line 24, in <module>
    import botocore.utils
  File "/usr/lib/fence-agents/bundled/botocore/utils.py", line 32, in <module>
    import botocore.httpsession
  File "/usr/lib/fence-agents/bundled/botocore/httpsession.py", line 10, in <module>
    from urllib3.util.ssl_ import (
ImportError: cannot import name PROTOCOL_TLS
2022-04-07 12:10:52,605:ERROR:certbot._internal.log:An unexpected error occurred:
2022-04-07 12:10:52,605:ERROR:certbot._internal.log:ImportError: cannot import name PROTOCOL_TLS

I raised in https://bugzilla.redhat.com/show_bug.cgi?id=2072990
Comment 16 Jered Floyd 2022-04-19 03:28:49 UTC 
At least on CentOS 7 (have not yet tested on RHEL 7) this breaks the currently packaged version of awscli (awscli-1.14.28-5.el7_5.1):

# aws --version
Traceback (most recent call last):
  File "/bin/aws", line 19, in <module>
    import awscli.clidriver
  File "/usr/lib/python2.7/site-packages/awscli/clidriver.py", line 37, in <module>
    from awscli.help import ProviderHelpCommand
  File "/usr/lib/python2.7/site-packages/awscli/help.py", line 24, in <module>
    from botocore.docs.bcdoc import docevents
ImportError: cannot import name docevents