I think there may actually be two issues here.

On the ansible controller, you need some or all of these modules present for the tasks to run.  Since ansible-core is 3.8, they must be present in python 3.8.

On the target host, system python libraries like selinux and firewall(d) should belong to platform-python, rather than any particular user-installed version of python.  However, when python38 is installed on a target machine, ansible chooses that as the interpreter over the system platform-python in either auto or auto_legacy mode.  To me, it seems like the client side fix would be for ansible to prefer /usr/libexec/platform-python over any version in /usr/bin.

> Ansible 5.4 requires Python 3.8, and some of the commonly used modules in Ansible fail due to missing Python libraries that are available in Python3 (3.6), but have no python 3.8 equivalents (yet)

It looks like there's a confusion between the python version mandatory by the ansible controller starting 2.12 (python 3.8+) and the one used by the modules on remote nodes (python 2.7 and 3.6+). Only plugins (lookup/callback or so) need to use python 3.8 since there are executed on the ansible controller unlike modules.

As long as the discovered python interpreter is picking up the right one (or force it the ansible/host configuration [1]) then there's no need to provide python modules dependencies (dnf, firewall, selinux, etc...) for python 3.8 and you can still use the python modules built with the default python interpreter (3.6 on stream8 via the python3-xxx packages).

I tested this on two Centos Stream 8 nodes, with both the default platform python (3.6) and python 3.8 installed.

one node is running the ansible controller and the other node is a remote node.

---------------------------------------------------
# python3 -V
Python 3.8.12
# cat inventory 
hostname.example.com ansible_connection=local
hostname2.example.com ansible_user=cloud-user ansible_host=x.x.x.x
# ansible -vv -i inventory all -m ping
ansible [core 2.12.5]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.8.12 (default, Sep 21 2021, 00:10:52) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]
  jinja version = 2.10.3
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
META: ran handlers
hostname.example.com | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
hostname2.example.com | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
META: ran handlers
META: ran handlers
---------------------------------------------------

The controller is correctly reporting that 3.8 is used but even if the default python3 interpreter is 3.8 on both nodes then ansible is correctly using /usr/libexec/platform-python (3.6) for remote execution.

For instance running `ansible -vv -i inventory all -b -m dnf -a 'name=bash state=present'` works perfectly even if python38-dnf (it doesn't exist anyway) isn't installed on both nodes because the discovered interpreter is 3.6.

Finally, the scope of the ansible-core package in RHEL is only limited to the rhel-system-role package and we don't provide extra python 3.8 for module dependencies.


[1] https://docs.ansible.com/ansible-core/2.12/reference_appendices/interpreter_discovery.html

I've been testing on Rocky 8.6.  I initially suspected that your newer version of ansible-core in 8-Stream (2.12.5 vs 2.12.2) fixed the problem, so I tried installing it with no change...except a new warning about the discovered python interpreter:

# ansible -m setup testhost -a filter=ansible_python
[WARNING]: Platform linux on host testhost is using the discovered Python interpreter at /usr/bin/python3.8, but future installation of another
Python interpreter could change the meaning of that path. See https://docs.ansible.com/ansible-core/2.12/reference_appendices/interpreter_discovery.html for
more information.
testhost | SUCCESS => {
    "ansible_facts": {
        "ansible_python": {
            "executable": "/usr/bin/python3.8",
            "has_sslcontext": true,
            "type": "cpython",
            "version": {
                "major": 3,
                "micro": 12,
                "minor": 8,
                "releaselevel": "final",
                "serial": 0
            },
            "version_info": [
                3,
                8,
                12,
                "final",
                0
            ]
        },
        "discovered_interpreter_python": "/usr/bin/python3.8"
    },
    "changed": false
}

After some further testing, I've discovered this doesn't happen on CentOS 8.4, but does happen on Rocky 8.5 (we switched to Rocky as of 8.5).  I don't have any other VM templates handy to test at the moment, but either something changed between 8.4 and 8.5, or this issue is specific to Rocky.  Should I open a new ticket for this against ansible since this one has been closed?

The issue here is that ansible 2.12 discovers the correct system Python interpreter (/usr/libexec/platform-python) on some, but not all, of the EL distributions. The Python interpreter discovery code has a hard coded list of EL distributions instead of determining them based on ansible_os_family. This was fixed upstream in https://github.com/ansible/ansible/pull/76815. Can the maintainers please backport this to ansible-core in RHEL 8 and 9? I don't think it can be backported upstream, because `stable-2.12` is in critical/security bugfix only mode.

> This was fixed upstream in https://github.com/ansible/ansible/pull/76815. Can the maintainers please backport this to ansible-core in RHEL 8 and 9?

I've proposed a c8s MR: https://gitlab.com/redhat/centos-stream/rpms/ansible-core/-/merge_requests/16

(In reply to Maxwell G from comment #5)
> > This was fixed upstream in https://github.com/ansible/ansible/pull/76815. Can the maintainers please backport this to ansible-core in RHEL 8 and 9?
> 
> I've proposed a c8s MR:
> https://gitlab.com/redhat/centos-stream/rpms/ansible-core/-/merge_requests/16

@dsavinea, please let me know if there's anything that I need to do to move this process forward.

> After some further testing, I've discovered this doesn't happen on CentOS 8.4, but does happen on Rocky 8.5 (we switched to Rocky as of 8.5).  I don't have any other VM templates handy to test at the moment, but either something changed between 8.4 and 8.5, or this issue is specific to Rocky.

So I made some extra testing on this and this isn't related to a specific ansible-core 2.12.x version but something specific to Rocky

-------------------------------
# cat inventory 
rhel86 ansible_user=cloud-user ansible_host=x.x.x.x ansible_become=true
rocky86 ansible_user=rocky ansible_host=x.x.x.x ansible_become=true
centosstream8 ansible_user=centos ansible_host=x.x.x.x ansible_become=true
centoslinux86 ansible_user=centos ansible_host=x.x.x.x ansible_become=true# ansible -vv -i inventory all -m ping
ansible [core 2.12.2]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.8.12 (default, Sep 16 2021, 10:46:05) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]
  jinja version = 2.10.3
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
META: ran handlers
rhel86 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
centosstream8 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
[WARNING]: Platform linux on host rocky86 is using the discovered Python interpreter at /usr/bin/python3.8, but future installation of another Python interpreter could change the meaning of that path. See
https://docs.ansible.com/ansible-core/2.12/reference_appendices/interpreter_discovery.html for more information.
rocky86 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3.8"
    },
    "changed": false,
    "ping": "pong"
}
centoslinux86 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
META: ran handlers
META: ran handlers
-------------------------------

This works for all 8.6 releases (RHEL/CentOS Stream/CentOS Linux) expect Rocky.

> The issue here is that ansible 2.12 discovers the correct system Python interpreter (/usr/libexec/platform-python) on some, but not all, of the EL distributions. The Python interpreter discovery code has a hard oded list of EL distributions instead of determining them based on ansible_os_family. This was fixed upstream in https://github.com/ansible/ansible/pull/76815.

This isn't correct. Rocky is missing in the interpreter python distro map variable.
It has been fixed in [1] via [2] but only included in 2.13.0+
The PR you mentioned is only about refactoring.
Once [1] is applied then the Rocky node uses the system python by default.

-------------------------------
# ansible -vv -i inventory all -m ping
ansible [core 2.12.2]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.8.12 (default, Sep 16 2021, 10:46:05) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]
  jinja version = 2.10.3
  libyaml = True
Using /etc/ansible/ansible.cfg as config file
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.
META: ran handlers
centoslinux86 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
rhel86 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
centosstream8 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
rocky86 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
META: ran handlers
META: ran handlers
-------------------------------

> Can the maintainers please backport this to ansible-core in RHEL 8 and 9? I don't think it can be backported upstream, because `stable-2.12` is in critical/security bugfix only mode.

This probably won't make it in the stable-2.12 branch upstream but I don't see why this needs to merge included in the current ansible-core package.
The current behaviour can be easily workaround by:

1/ adding "ansible_python_interpreter=/usr/libexec/platform-python" to the Rocky nodes in the inventory, host_vars or group_vars (if all nodes aren't Rocky based)
2/ adding "interpreter_python = /usr/libexec/platform-python" in the ansible.cfg file under the defaults section (note that this will be global to all nodes)

> I've proposed a c8s MR: https://gitlab.com/redhat/centos-stream/rpms/ansible-core/-/merge_requests/16

That's not the right workflow for ansible-core changes in RHEL/CentOS Stream 8.
The changes are done in RHEL 8 first and then synced back to CentOS 8 Stream repository (unlike RHEL/CentOS Stream 9, where changes land first in Stream).
Finally the patch doesn't fix the issue at all.
You'll need to apply [3] via [4] on top of it.

As I said, considering this is a specific Rocky issue that can be fixed by some extra configuration (described in [5]) and we plan to upgrade to 2.13 in a near future then I don't see the point of adding extra patches to the current package configuration (which aren't upstream on the stable-2.12 branch)

[1] https://github.com/ansible/ansible/commit/f2c8b59eb17b0a432b397e0e79afcaf7495930c9
[2] https://github.com/ansible/ansible/pull/76728
[3] https://github.com/ansible/ansible/commit/4723eb9caa12e1b99c4c411199bd4d4ab272534d
[4] https://github.com/ansible/ansible/pull/77371
[5] https://docs.ansible.com/ansible-core/2.12/reference_appendices/interpreter_discovery.html

> So I made some extra testing on this and this isn't related to a specific
> ansible-core 2.12.x version 

Yes, I came to the same conclusion.

> but something specific to Rocky

Well, it's not just Rocky. It also applies to Almalinux and other RHEL rebuilds. This is why I backported the refactoring the patch which fixes all the EL distributions instead of just backporting [1].

> > Can the maintainers please backport this to ansible-core in RHEL 8 and 9? I don't think it can be backported upstream, because `stable-2.12` is in critical/security bugfix only mode.
> 
> This probably won't make it in the stable-2.12 branch upstream but I don't
> see why this needs to merge included in the current ansible-core package.
> The current behaviour can be easily workaround by:
> 
> 1/ adding "ansible_python_interpreter=/usr/libexec/platform-python" to the
> Rocky nodes in the inventory, host_vars or group_vars (if all nodes aren't
> Rocky based)
> 2/ adding "interpreter_python = /usr/libexec/platform-python" in the
> ansible.cfg file under the defaults section (note that this will be global
> to all nodes)

It's needed, because users are facing issues with those distributions and opening bugs asking you and me (Fedora/EPEL ansible and ansible-collection-* co-maintainer) to package libraries for python38. I have already been telling users to do 1/, but this should work properly out of the box.

> > I've proposed a c8s MR: https://gitlab.com/redhat/centos-stream/rpms/ansible-core/-/merge_requests/16
> 
> That's not the right workflow for ansible-core changes in RHEL/CentOS Stream
> 8.
> The changes are done in RHEL 8 first and then synced back to CentOS 8 Stream
> repository (unlike RHEL/CentOS Stream 9, where changes land first in Stream).

Can you please point me in the right direction for the future? Where should we direct patches for EL 8? This really should be documented properly.

> Finally the patch doesn't fix the issue at all.

Well, it fixed the issue for Rocky and Alma. It looks like @sivel accidentally broke RHEL when they introduced it which they fixed in [4].

> We plan to upgrade to 2.13 in a near future

When is this currently planned for? https://access.redhat.com/support/policy/updates/rhel-app-streams-life-cycle claims ansible-core will be supported in RHEL 8 and 9 until Nov 2023, but it doesn't provide any information about which version or what will happen after that; I would be surprised if ansible-core is only be supported for the first year and a half of RHEL 9's life cycle.

> 
> [1]
> https://github.com/ansible/ansible/commit/
> f2c8b59eb17b0a432b397e0e79afcaf7495930c9
> [2] https://github.com/ansible/ansible/pull/76728
> [3]
> https://github.com/ansible/ansible/commit/
> 4723eb9caa12e1b99c4c411199bd4d4ab272534d
> [4] https://github.com/ansible/ansible/pull/77371
> [5]
> https://docs.ansible.com/ansible-core/2.12/reference_appendices/
> interpreter_discovery.html

(In reply to Maxwell G from comment #8)
> 
> > We plan to upgrade to 2.13 in a near future
> 
> When is this currently planned for?
> https://access.redhat.com/support/policy/updates/rhel-app-streams-life-cycle
> claims ansible-core will be supported in RHEL 8 and 9 until Nov 2023, but it
> doesn't provide any information about which version or what will happen
> after that; I would be surprised if ansible-core is only be supported for
> the first year and a half of RHEL 9's life cycle.

I would appreciate if you could provide more information here, if possible.

> > > I've proposed a c8s MR: https://gitlab.com/redhat/centos-stream/rpms/ansible-core/-/merge_requests/16
> > 
> > That's not the right workflow for ansible-core changes in RHEL/CentOS Stream
> > 8.
> > The changes are done in RHEL 8 first and then synced back to CentOS 8 Stream
> > repository (unlike RHEL/CentOS Stream 9, where changes land first in Stream).
> 
> Can you please point me in the right direction for the future? Where should
> we direct patches for EL 8? This really should be documented properly.

I am also still interested in this. My understanding is that it's possible to submit patches against c8s, but the maintainers have to manually apply it to RHEL dist-git. At least this is what I gathered from working with the python-maint team ([1] and [2], for example). Given that CentOS Stream is supposed to make community contribution easier, there should be a clear way to contribute to it.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=2086141
[2]: https://git.centos.org/rpms/python-jinja2/c/5d802ed3f471da01d494e73569acd3b2b181ce0c?branch=c8s-stream-3.8

> Can you please point me in the right direction for the future? Where should we direct patches for EL 8? This really should be documented properly.

It's in the RHEL dist-git repository which is not accessible publicly.

> I am also still interested in this. My understanding is that it's possible to submit patches against c8s, but the maintainers have to manually apply it to RHEL dist-git. At least this is what I gathered from working with the python-maint team ([1] and [2], for example). Given that CentOS Stream is supposed to make community contribution easier, there should be a clear way to contribute to it.

Patches on c8s aren't accepted "as is", meaning if there are accepted, maintainers need to commit the changes in RHEL 8 dist-git first but nothing is merged directly on the c8s branch.

That's exactly what happened with your jinja2 patch. Commited in the dist-git by maintainers and then synced back automatically on the c8s-stream-3.8 branch (since that's a package from the python 3.8 stream)

There's no sync c8s -> RHEL 8 (hence we can't accept PR). It's only RHEL 8 to c8s.

https://git.centos.org/rpms/ansible-core/commits/c8s

Don't get me wrong, I'm not saying that patches against c8s will always be rejected. I'm saying that there's no point of submitting them as PR on the c8s gitlab branch (that's only true starting with c9s).
I guess the workflow for that is to propose patches in a BZ like you did for jinja2.

> When is this currently planned for?

It's scheduled for the upcoming weeks.

> https://access.redhat.com/support/policy/updates/rhel-app-streams-life-cycle claims ansible-core will be supported in RHEL 8 and 9 until Nov 2023, but it doesn't provide any information about which version or what will happen after that; I would be surprised if ansible-core is only be supported for the first year and a half of RHEL 9's life cycle.

This date is only for RHEL 8.6 and 9.0 (ie: ansible 2.12) which won't be affected by the ansible-core update (ie: 8.7 and 9.1)
Basically on each RHEL Y releases, the ansible-core package will be updated.

For anyone who comes across this bugzilla, Rocky Linux has fixed the Python interpreter discovery issue downstream with https://git.rockylinux.org/staging/rpms/ansible-core/-/commit/d160ce8b36de0d4adfba55a6fc9ff5d9f421bb31.