Description of problem:
OSP16.1 deployment fails during overcloud deploy step on ceph-ansible part with error:

# undercloud-0/var/lib/mistral/overcloud/ansible.log.txt.gz

2020-05-04 18:06:56,886 p=730 u=mistral n=ansible | TASK [tripleo-ceph-run-ansible : print ceph-ansible output in case of failure] ***
2020-05-04 18:06:56,886 p=730 u=mistral n=ansible | Monday 04 May 2020  18:06:56 +0000 (0:00:00.246)       0:17:37.490 ************ 
2020-05-04 18:06:56,948 p=730 u=mistral n=ansible | fatal: [undercloud]: FAILED! => {
    "ceph_ansible_std_out_err": [
...
2020-05-04 18:06:56,958 p=730 u=mistral n=ansible | undercloud                 : ok=62   changed=23   unreachable=0    failed=1    skipped=38   rescued=0    ignored=0

# undercloud-0/var/lib/mistral/overcloud/ceph-ansible/ceph_ansible_command.log.txt.gz

2020-05-04 18:06:56,242 p=181609 u=root n=ansible | TASK [ceph-validate : fail on unsupported ansible version] *********************
2020-05-04 18:06:56,242 p=181609 u=root n=ansible | task path: /usr/share/ceph-ansible/roles/ceph-validate/tasks/check_system.yml:61
2020-05-04 18:06:56,242 p=181609 u=root n=ansible | Monday 04 May 2020  18:06:56 +0000 (0:00:00.115)       0:00:26.958 ************ 
2020-05-04 18:06:56,285 p=181609 u=root n=ansible | fatal: [controller-0]: FAILED! => changed=false 
  msg: Ansible version must be 2.8!
2020-05-04 18:06:56,324 p=181609 u=root n=ansible | fatal: [ceph-0]: FAILED! => changed=false 
  msg: Ansible version must be 2.8!
2020-05-04 18:06:56,342 p=181609 u=root n=ansible | fatal: [compute-0]: FAILED! => changed=false

Problem seems to be newer ansible version on OC nodes:
ansible.noarch                                2.9.7-1.el8ae                                   @rhos-16.1-rhel-8-ansible    

Version:
OSP16.1, appears since puddle RHOS-16.1-RHEL-8-20200504.n.0

Additional information:
might be related to upstream's Centos 8 issue https://bugs.launchpad.net/tripleo/+bug/1864771
logs attached privately