Bug 2252782 - "TypeError: can't compare offset-naive and offset-aware datetimes" when awscli2 is fetching credentials via iam-role
Summary: "TypeError: can't compare offset-naive and offset-aware datetimes" when awscl...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: awscli2
Version: 39
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: Nikola Forró
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-12-04 16:37 UTC by Chris Palmer
Modified: 2023-12-20 01:25 UTC (History)
2 users (show)

Fixed In Version: awscli2-2.15.1-1.fc39
Clone Of:
Environment:
Last Closed: 2023-12-20 01:25:06 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Chris Palmer 2023-12-04 16:37:51 UTC
Using awscli2 in conjunction with IAM Role credentials fails with a date-related error under FC39. It worked correctly under FC38.

Reproducible: Always

Steps to Reproduce:
1.Create an EC2 instance of Fedora 39 Cloud 
2.Assign an IAM role to the instance
3.aws s3 ls
Actual Results:  
aws s3 ls --debug
2023-12-04 16:25:42,672 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.13.7 Python/3.12.0 Linux/6.6.3-200.fc39.x86_64 source/x86_64.fedora.39
2023-12-04 16:25:42,673 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'ls', '--debug']
2023-12-04 16:25:42,702 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7ff685320cc0>
2023-12-04 16:25:42,702 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7ff685560b80>
2023-12-04 16:25:42,702 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2023-12-04 16:25:42,702 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7ff6856ded40>
2023-12-04 16:25:42,702 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7ff6856f44a0>
2023-12-04 16:25:42,702 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7ff685323420>
2023-12-04 16:25:42,702 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7ff6855a3b00>
2023-12-04 16:25:42,703 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2023-12-04 16:25:42,703 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7ff6853232e0>
2023-12-04 16:25:42,703 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7ff68538a060>>
2023-12-04 16:25:42,704 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/lib/python3.12/site-packages/awscli/data/cli.json
2023-12-04 16:25:42,707 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7ff6854716c0>
2023-12-04 16:25:42,707 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7ff6854719e0>
2023-12-04 16:25:42,707 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7ff685471940>
2023-12-04 16:25:42,708 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7ff685471b20>
2023-12-04 16:25:42,708 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7ff685471a80>
2023-12-04 16:25:42,708 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7ff6853d1800>
2023-12-04 16:25:42,708 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.13.7 Python/3.12.0 Linux/6.6.3-200.fc39.x86_64 source/x86_64.fedora.39 prompt/off
2023-12-04 16:25:42,708 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'ls', '--debug']
2023-12-04 16:25:42,709 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7ff685321440>
2023-12-04 16:25:42,709 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7ff685709c60>
2023-12-04 16:25:42,709 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7ff68538fec0>
2023-12-04 16:25:42,709 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7ff685d245e0>
2023-12-04 16:25:42,709 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7ff6857198a0>
2023-12-04 16:25:42,711 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2023-12-04 16:25:42,713 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7ff6855a0220>
2023-12-04 16:25:42,713 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7ff685558fe0>
2023-12-04 16:25:42,713 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7ff6853232e0>
2023-12-04 16:25:42,713 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7ff68538a060>>
2023-12-04 16:25:42,714 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_ls: calling handler <function add_waiters at 0x7ff6853232e0>
2023-12-04 16:25:42,714 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_ls: calling handler <bound method AliasSubCommandInjector.on_building_command_table of <awscli.alias.AliasSubCommandInjector object at 0x7ff68538a060>>
2023-12-04 16:25:42,716 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff685cdd7c0>
2023-12-04 16:25:42,716 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff685cdd7c0>
2023-12-04 16:25:42,716 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff685cdd7c0>
2023-12-04 16:25:42,716 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.human-readable: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff685cdd7c0>
2023-12-04 16:25:42,716 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ff6854d3440>
2023-12-04 16:25:42,716 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.summarize: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff685cdd7c0>
2023-12-04 16:25:42,716 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7ff6854d3440>
2023-12-04 16:25:42,716 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.request-payer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7ff685cdd7c0>
2023-12-04 16:25:42,717 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2023-12-04 16:25:42,717 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2023-12-04 16:25:42,717 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2023-12-04 16:25:42,717 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2023-12-04 16:25:42,717 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2023-12-04 16:25:42,717 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: custom-process
2023-12-04 16:25:42,717 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: config-file
2023-12-04 16:25:42,718 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: ec2-credentials-file
2023-12-04 16:25:42,719 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: boto-config
2023-12-04 16:25:42,719 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: container-role
2023-12-04 16:25:42,719 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: iam-role
2023-12-04 16:25:42,720 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 169.254.169.254:80
2023-12-04 16:25:42,721 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "PUT /latest/api/token HTTP/1.1" 200 56
2023-12-04 16:25:42,722 - MainThread - urllib3.connectionpool - DEBUG - Resetting dropped connection: 169.254.169.254
2023-12-04 16:25:42,723 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 5
2023-12-04 16:25:42,724 - MainThread - urllib3.connectionpool - DEBUG - Resetting dropped connection: 169.254.169.254
2023-12-04 16:25:42,725 - MainThread - urllib3.connectionpool - DEBUG - http://169.254.169.254:80 "GET /latest/meta-data/iam/security-credentials/peweb HTTP/1.1" 200 1570
2023-12-04 16:25:42,729 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/awscli/clidriver.py", line 460, in main
    return command_table[parsed_args.command](remaining, parsed_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/customizations/commands.py", line 151, in __call__
    return self._subcommand_table[subcommand_name](
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/customizations/commands.py", line 205, in __call__
    rc = self._run_main(parsed_args, parsed_globals)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/customizations/s3/subcommands.py", line 494, in _run_main
    super(ListCommand, self)._run_main(parsed_args, parsed_globals)
  File "/usr/lib/python3.12/site-packages/awscli/customizations/s3/subcommands.py", line 480, in _run_main
    self.client = ClientFactory(self._session).create_client(params)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/customizations/s3/factory.py", line 50, in create_client
    return self._session.create_client('s3', **create_client_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/botocore/session.py", line 839, in create_client
    credentials = self.get_credentials()
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/botocore/session.py", line 445, in get_credentials
    'credential_provider').load_credentials()
                           ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/botocore/credentials.py", line 1943, in load_credentials
    creds = provider.load()
            ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/botocore/credentials.py", line 996, in load
    metadata = fetcher.retrieve_iam_role_credentials()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/awscli/botocore/utils.py", line 500, in retrieve_iam_role_credentials
    self._evaluate_expiration(credentials)
  File "/usr/lib/python3.12/site-packages/awscli/botocore/utils.py", line 583, in _evaluate_expiration
    if current_time >= extension_time:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: can't compare offset-naive and offset-aware datetimes

can't compare offset-naive and offset-aware datetimes


Expected Results:  
aws command should retrieve the IAM role credentials successfully

Another user has already logged this is in github for botocore:

https://github.com/boto/botocore/issues/3077

python3-botocore is 1.31.21-1 on both FC38 & FC39. It is not installed by default on FC39, however manually installing it has no effect.

awscli2 is 2.13.4-1 on FC38, and 2.13.7-1 on FC39.

Comment 1 Nikola Forró 2023-12-06 18:49:10 UTC
Thanks for the report.

> python3-botocore is 1.31.21-1 on both FC38 & FC39. It is not installed by default on FC39, however manually installing it has no effect.

That's expected, aws-cli bundles its own copy of botocore, so python3-botocore is not used.

Could you try installing rawhide package [1] to see if it fixes the problem? I believe Python 3.12 issues should be sorted out there.

[1] https://koji.fedoraproject.org/koji/buildinfo?buildID=2323310

Comment 2 Fedora Update System 2023-12-15 16:26:40 UTC
FEDORA-2023-2f7758fdda has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-2f7758fdda

Comment 3 Nikola Forró 2023-12-15 16:29:03 UTC
I've rebased to the latest upstream version and fixed another bunch of Python 3.12 issues. I believe it should fix also this one, please let me know if it doesn't.

Comment 4 Fedora Update System 2023-12-16 01:05:41 UTC
FEDORA-2023-2f7758fdda has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-2f7758fdda`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-2f7758fdda

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 5 Fedora Update System 2023-12-20 01:25:06 UTC
FEDORA-2023-2f7758fdda has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.