Bug 2252782

Summary: "TypeError: can't compare offset-naive and offset-aware datetimes" when awscli2 is fetching credentials via iam-role
Product: [Fedora] Fedora Reporter: Chris Palmer <chris.palmer>
Component: awscli2Assignee: Nikola Forró <nforro>
Status: CLOSED ERRATA QA Contact:
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 39CC: davdunc, nforro
Target Milestone: ---Keywords: Regression
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: awscli2-2.15.1-1.fc39 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-12-20 01:25:06 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Chris Palmer 2023-12-04 16:37:51 UTC 
Using awscli2 in conjunction with IAM Role credentials fails with a date-related error under FC39. It worked correctly under FC38.

Reproducible: Always

Steps to Reproduce:
1.Create an EC2 instance of Fedora 39 Cloud 
2.Assign an IAM role to the instance
3.aws s3 ls
Actual Results:  
aws s3 ls --debug
2023-12-04 16:25:42,672 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.13.7 Python/3.12.0 Linux/6.6.3-200.fc39.x86_64 source/x86_64.fedora.39
2023-12-04 16:25:42,673 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'ls', '--debug']
2023-12-04 16:25:42,702 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7ff685320cc0>
2023-12-04 16:25:42,702 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7ff685560b80>
2023-12-04 16:25:42,702 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2023-12-04 16:25:42,702 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7ff6856ded40>
2023-12-04 16:25:42,702 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7ff6856f44a0>
2023-12-04 16:25:42,702 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7ff685323420>
2023-12-04 16:25:42,702 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7ff6855a3b00>
2023-12-04 16:25:42,703 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2023-12-04 16:25:42,703 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7ff6853232e0>
2023-12-04 16:25:42,703 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7ff68538a060>>
2023-12-04 16:25:42,704 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python3.12/site-packages/awscli/data/cli.json
2023-12-04 16:25:42,707 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7ff6854716c0>
2023-12-04 16:25:42,707 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7ff6854719e0>
2023-12-04 16:25:42,707 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7ff685471940>
2023-12-04 16:25:42,708 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7ff685471b20>
2023-12-04 16:25:42,708 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7ff685471a80>
2023-12-04 16:25:42,708 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7ff6853d1800>
2023-12-04 16:25:42,708 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.13.7 Python/3.12.0 Linux/6.6.3-200.fc39.x86_64 source/x86_64.fedora.39 prompt/off
2023-12-04 16:25:42,708 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'ls', '--debug']
2023-12-04 16:25:42,709 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7ff685321440>
2023-12-04 16:25:42,709 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7ff685709c60>
2023-12-04 16:25:42,709 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7ff68538fec0>
2023-12-04 16:25:42,709 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7ff685d245e0>
2023-12-04 16:25:42,709 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7ff6857198a0>
2023-12-04 16:25:42,711 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2023-12-04 16:25:42,713 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7ff6855a0220>
2023-12-04 16:25:42,713 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7ff685558fe0>
2023-12-04 16:25:42,713 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7ff6853232e0>
2023-12-04 16:25:42,713 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7ff68538a060>>
2023-12-04 16:25:42,714 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_ls: calling handler <function add_waiters at 0x7ff6853232e0>
2023-12-04 16:25:42,714 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_ls: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7ff68538a060>>
2023-12-04 16:25:42,716 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff685cdd7c0>
2023-12-04 16:25:42,716 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff685cdd7c0>
2023-12-04 16:25:42,716 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff685cdd7c0>
2023-12-04 16:25:42,716 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.human-readable: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff685cdd7c0>
2023-12-04 16:25:42,716 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ff6854d3440>
2023-12-04 16:25:42,716 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.summarize: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff685cdd7c0>
2023-12-04 16:25:42,716 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ff6854d3440>
2023-12-04 16:25:42,716 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.request-payer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff685cdd7c0>
2023-12-04 16:25:42,717 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2023-12-04 16:25:42,717 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2023-12-04 16:25:42,717 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2023-12-04 16:25:42,717 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2023-12-04 16:25:42,717 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2023-12-04 16:25:42,717 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: custom-process
2023-12-04 16:25:42,717 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: config-file
2023-12-04 16:25:42,718 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: ec2-credentials-file
2023-12-04 16:25:42,719 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: boto-config
2023-12-04 16:25:42,719 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: container-role
2023-12-04 16:25:42,719 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: iam-role
2023-12-04 16:25:42,720 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 169.254.169.254:80
2023-12-04 16:25:42,721 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "PUT /latest/api/token HTTP/1.1" 200 56
2023-12-04 16:25:42,722 - MainThread - urllib3.connectionpool - DEBUG - Resetting dropped connection: 169.254.169.254
2023-12-04 16:25:42,723 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 5
2023-12-04 16:25:42,724 - MainThread - urllib3.connectionpool - DEBUG - Resetting dropped connection: 169.254.169.254
2023-12-04 16:25:42,725 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/peweb HTTP/1.1" 200 1570
2023-12-04 16:25:42,729 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/awscli/clidriver.py", line 460, in main
    return command_table[parsed_args.command](remaining, parsed_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/customizations/commands.py", line 151, in __call__
    return self._subcommand_table[subcommand_name](
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/customizations/commands.py", line 205, in __call__
    rc = self._run_main(parsed_args, parsed_globals)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/customizations/s3/subcommands.py", line 494, in _run_main
    super(ListCommand, self)._run_main(parsed_args, parsed_globals)
  File "/usr/lib/python3.12/site-packages/awscli/customizations/s3/subcommands.py", line 480, in _run_main
    self.client = ClientFactory(self._session).create_client(params)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/customizations/s3/factory.py", line 50, in create_client
    return self._session.create_client('s3', **create_client_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/botocore/session.py", line 839, in create_client
    credentials = self.get_credentials()
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/botocore/session.py", line 445, in get_credentials
    'credential_provider').load_credentials()
                           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/botocore/credentials.py", line 1943, in load_credentials
    creds = provider.load()
            ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/botocore/credentials.py", line 996, in load
    metadata = fetcher.retrieve_iam_role_credentials()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/botocore/utils.py", line 500, in retrieve_iam_role_credentials
    self._evaluate_expiration(credentials)
  File "/usr/lib/python3.12/site-packages/awscli/botocore/utils.py", line 583, in _evaluate_expiration
    if current_time >= extension_time:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: can't compare offset-naive and offset-aware datetimes

can't compare offset-naive and offset-aware datetimes


Expected Results:  
aws command should retrieve the IAM role credentials successfully

Another user has already logged this is in github for botocore:

https://github.com/boto/botocore/issues/3077

python3-botocore is 1.31.21-1 on both FC38 & FC39. It is not installed by default on FC39, however manually installing it has no effect.

awscli2 is 2.13.4-1 on FC38, and 2.13.7-1 on FC39.
Comment 1 Nikola Forró 2023-12-06 18:49:10 UTC 
Thanks for the report.

> python3-botocore is 1.31.21-1 on both FC38 & FC39. It is not installed by default on FC39, however manually installing it has no effect.

That's expected, aws-cli bundles its own copy of botocore, so python3-botocore is not used.

Could you try installing rawhide package [1] to see if it fixes the problem? I believe Python 3.12 issues should be sorted out there.

[1] https://koji.fedoraproject.org/koji/buildinfo?buildID=2323310
Comment 2 Fedora Update System 2023-12-15 16:26:40 UTC 
FEDORA-2023-2f7758fdda has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-2f7758fdda
Comment 3 Nikola Forró 2023-12-15 16:29:03 UTC 
I've rebased to the latest upstream version and fixed another bunch of Python 3.12 issues. I believe it should fix also this one, please let me know if it doesn't.
Comment 4 Fedora Update System 2023-12-16 01:05:41 UTC 
FEDORA-2023-2f7758fdda has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-2f7758fdda`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-2f7758fdda

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 5 Fedora Update System 2023-12-20 01:25:06 UTC 
FEDORA-2023-2f7758fdda has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.