Was testing an upgrade from 13 to 16.1 and ran the following command:

openstack overcloud external-upgrade run --tags ceph_systemd -e ceph_ansible_limit=overcloud-controller-0

I hit these errors:

TASK [ceph : Get ceph-ansible repository] **************************************
Wednesday 24 June 2020  23:19:00 +1000 (0:00:00.057)       0:00:18.191 ******** 
[WARNING]: Consider using the yum module rather than running 'yum'.  If you
need to use command because yum is insufficient you can add 'warn: false' to
this command task or set 'command_warnings=False' in ansible.cfg to get rid of
this message.
ok: [undercloud] => {"changed": false, "cmd": "yum info ceph-ansible | awk '/From repo/ {print $4}'", "delta": "0:00:01.264995", "end": "2020-06-24 23:19:02.103025", "rc": 0, "start": "2020-06-24 23:19:00.838030", "stderr": "2020-06-24 23:19:01,374 [ERROR] yum:981965:MainThread @logutil.py:194 - [Errno 13] Permission denied: '/var/log/rhsm/rhsm.log' - Further logging output will be written to stderr\n/usr/lib/python3.6/site-packages/dateutil/parser/_parser.py:70: UnicodeWarning: decode() called on unicode string, see https://bugzilla.redhat.com/show_bug.cgi?id=1693751\n  instream = instream.decode()\n\nErrors during downloading metadata for repository 'openstack-beta-for-rhel-8-x86_64-rpms':\n  - Curl error (58): Problem with the local SSL certificate for https://cdn.redhat.com/content/beta/layered/rhel8/x86_64/openstack/os/repodata/repomd.xml [unable to set private key file: '/etc/pki/entitlement/712953222321075321-key.pem' type PEM]\nError: Failed to download metadata for repo 'openstack-beta-for-rhel-8-x86_64-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", "stderr_lines": ["2020-06-24 23:19:01,374 [ERROR] yum:981965:MainThread @logutil.py:194 - [Errno 13] Permission denied: '/var/log/rhsm/rhsm.log' - Further logging output will be written to stderr", "/usr/lib/python3.6/site-packages/dateutil/parser/_parser.py:70: UnicodeWarning: decode() called on unicode string, see https://bugzilla.redhat.com/show_bug.cgi?id=1693751", "  instream = instream.decode()", "", "Errors during downloading metadata for repository 'openstack-beta-for-rhel-8-x86_64-rpms':", "  - Curl error (58): Problem with the local SSL certificate for https://cdn.redhat.com/content/beta/layered/rhel8/x86_64/openstack/os/repodata/repomd.xml [unable to set private key file: '/etc/pki/entitlement/712953222321075321-key.pem' type PEM]", "Error: Failed to download metadata for repo 'openstack-beta-for-rhel-8-x86_64-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried"], "stdout": "", "stdout_lines": []}

TASK [ceph : Fail if ceph-ansible doesn't belong to the specified repo] ********
Wednesday 24 June 2020  23:19:02 +1000 (0:00:01.517)       0:00:19.708 ******** 
[WARNING]: conditional statements should not include jinja2 templating
delimiters such as {{ }} or {% %}. Found: (repo.stdout | length == 0 or
repo.stdout != "{{ ceph_ansible_repo }}")
fatal: [undercloud]: FAILED! => {"changed": false, "msg": "Make sure ceph-ansible package is installed from rhceph-4-tools-for-rhel-8-x86_64-rpms or configure the repo name you intend to install it from using the 'CephAnsibleRepo' variable provided by tripleo-heat-templates"}

It seems that running "yum info ceph-ansible" within the playbook causes yum/dnf to download the repo metadata but couldn't access the pem certificates used to access the CDN. I manually edited this validation to use "sudo yum info ceph-ansible" and it worked without issue.

So either we need to use sudo when running this command or better yet use "become: yes" to escalate the task.