Bug 2361828

Summary: [8.1][Cephadm-Ansible][cephadm-preflight.yml] Playbook fails at [Ensure reports directory exists on the Ansible controller]
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Sayalee <saraut>
Component: CephadmAssignee: Kushal Deb <kdeb>
Status: CLOSED ERRATA QA Contact: Sayalee <saraut>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 8.1CC: akane, cephqe-warriors, tserlin
Target Milestone: ---Keywords: Automation, Regression, TestBlocker
Target Release: 8.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: cephadm-ansible-4.1.4-1.el9cp Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2025-06-26 12:30:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sayalee 2025-04-23 09:07:51 UTC 
Description of problem:
========================

Cephadm preflight playbook is failing at task [Ensure reports directory exists on the Ansible controller]

Log snippet -
-----------
2025-04-23 11:42:24,977 - cephci - ceph:1186 - DEBUG - TASK [Ensure reports directory exists on the Ansible controller] ************
***                                                                                                                                 
2025-04-23 11:42:24,977 - cephci - ceph:1186 - DEBUG - task path: /usr/share/cephadm-ansible/checks.yml:231                         
2025-04-23 11:42:24,977 - cephci - ceph:1186 - DEBUG - Wednesday 23 April 2025  06:12:24 +0000 (0:00:00.237)       0:01:03.303 *****
**                                                                                                                                  
2025-04-23 11:42:25,489 - cephci - ceph:1186 - DEBUG - <localhost> ESTABLISH LOCAL CONNECTION FOR USER: cephuser                    
2025-04-23 11:42:25,490 - cephci - ceph:1186 - DEBUG - <localhost> EXEC /bin/sh -c 'echo ~cephuser && sleep 0'                      
2025-04-23 11:42:25,490 - cephci - ceph:1186 - DEBUG - <localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/cephuser/.a
nsible/tmp `"&& mkdir "` echo /home/cephuser/.ansible/tmp/ansible-tmp-1745388744.9345827-7930-154972548146525 `" && echo ansible-tmp
-1745388744.9345827-7930-154972548146525="` echo /home/cephuser/.ansible/tmp/ansible-tmp-1745388744.9345827-7930-154972548146525 `" 
) && sleep 0'                                                                                                                       
2025-04-23 11:42:25,491 - cephci - ceph:1186 - DEBUG - Using module file /usr/lib/python3.9/site-packages/ansible/modules/file.py   
2025-04-23 11:42:25,491 - cephci - ceph:1186 - DEBUG - <localhost> PUT /home/cephuser/.ansible/tmp/ansible-local-5986zl20yiqu/tmpka3
2qq8z TO /home/cephuser/.ansible/tmp/ansible-tmp-1745388744.9345827-7930-154972548146525/AnsiballZ_file.py                          
2025-04-23 11:42:25,491 - cephci - ceph:1186 - DEBUG - <localhost> EXEC /bin/sh -c 'chmod u+x /home/cephuser/.ansible/tmp/ansible-tm
p-1745388744.9345827-7930-154972548146525/ /home/cephuser/.ansible/tmp/ansible-tmp-1745388744.9345827-7930-154972548146525/AnsiballZ
_file.py && sleep 0'                                                                                                                
2025-04-23 11:42:25,491 - cephci - ceph:1186 - DEBUG - <localhost> EXEC /bin/sh -c '/usr/bin/python3 /home/cephuser/.ansible/tmp/ans
ible-tmp-1745388744.9345827-7930-154972548146525/AnsiballZ_file.py && sleep 0'                                                      
2025-04-23 11:42:25,491 - cephci - ceph:1186 - DEBUG - <localhost> EXEC /bin/sh -c 'rm -f -r /home/cephuser/.ansible/tmp/ansible-tmp
-1745388744.9345827-7930-154972548146525/ > /dev/null 2>&1 && sleep 0'                                                              
2025-04-23 11:42:25,492 - cephci - ceph:1186 - DEBUG - fatal: [ceph-saraut-rh-33mpct-node3 -> localhost]: FAILED! => changed=false 
2025-04-23 11:42:25,492 - cephci - ceph:1186 - DEBUG -   invocation:                                                                
2025-04-23 11:42:25,492 - cephci - ceph:1186 - DEBUG -     module_args:                                                             
2025-04-23 11:42:25,492 - cephci - ceph:1186 - DEBUG -       _diff_peek: null                                                       
2025-04-23 11:42:25,492 - cephci - ceph:1186 - DEBUG -       _original_basename: null                                               
2025-04-23 11:42:25,492 - cephci - ceph:1186 - DEBUG -       access_time: null                                                      
2025-04-23 11:42:25,492 - cephci - ceph:1186 - DEBUG -       access_time_format: '%Y%m%d%H%M.%S'                                    
2025-04-23 11:42:25,492 - cephci - ceph:1186 - DEBUG -       attributes: null
2025-04-23 11:42:25,493 - cephci - ceph:1186 - DEBUG -       follow: true
2025-04-23 11:42:25,493 - cephci - ceph:1186 - DEBUG -       force: false
2025-04-23 11:42:25,493 - cephci - ceph:1186 - DEBUG -       group: null
2025-04-23 11:42:25,493 - cephci - ceph:1186 - DEBUG -       mode: '0755'
2025-04-23 11:42:25,493 - cephci - ceph:1186 - DEBUG -       modification_time: null
2025-04-23 11:42:25,493 - cephci - ceph:1186 - DEBUG -       modification_time_format: '%Y%m%d%H%M.%S'
2025-04-23 11:42:25,493 - cephci - ceph:1186 - DEBUG -       owner: null
2025-04-23 11:42:25,493 - cephci - ceph:1186 - DEBUG -       path: ./reports
2025-04-23 11:42:25,493 - cephci - ceph:1186 - DEBUG -       recurse: false
2025-04-23 11:42:25,493 - cephci - ceph:1186 - DEBUG -       selevel: null
2025-04-23 11:42:25,494 - cephci - ceph:1186 - DEBUG -       serole: null
2025-04-23 11:42:25,494 - cephci - ceph:1186 - DEBUG -       setype: null
2025-04-23 11:42:25,494 - cephci - ceph:1186 - DEBUG -       seuser: null
2025-04-23 11:42:25,494 - cephci - ceph:1186 - DEBUG -       src: null
2025-04-23 11:42:25,494 - cephci - ceph:1186 - DEBUG -       state: directory
2025-04-23 11:42:25,495 - cephci - ceph:1186 - DEBUG -       unsafe_writes: false
2025-04-23 11:42:25,495 - cephci - ceph:1186 - DEBUG -   msg: 'There was an issue creating ./reports as requested: [Errno 13] Permission denied: b''./reports'''
2025-04-23 11:42:25,495 - cephci - ceph:1186 - DEBUG -   path: ./reports



Version-Release number of selected component (if applicable):
=============================================================
cephadm-ansible-4.1.3-1.el9cp.noarch


How reproducible:
=================
Always
All the suites running the preflight playbook are failing with same error.


Steps to Reproduce:
=====================
1. Install cephadm-anisble
2. Run cephadm-preflight playbook


Actual results:
================
cephadm-preflight playbook is failing at task [Ensure reports directory exists on the Ansible controller]

Expected results:
==================
cephadm-preflight should pass and should not throw error -
msg: 'There was an issue creating ./reports as requested: [Errno 13] Permission denied: b''./reports'''


Additional info:
================

When the playbook was executed manually as root user on the same installer node, it passed successfully

Command used -
cd /usr/share/cephadm-ansible; ansible-playbook -vvvv -i /usr/sha
re/cephadm-ansible/hosts cephadm-preflight.yml -e 'ceph_origin=custom' -e 'gpgcheck=no' -e 'custom_repo_url=http://download-01.beak-
001.prod.iad2.dc.redhat.com/rhel-9/composes/auto/ceph-8.1-rhel-9/RHCEPH-8.1-RHEL-9-20250422.ci.2/compose/Tools/x86_64/os'


Results -
PLAY RECAP *************************************************************************************************************************************************
ceph-saraut-rh-33mpct-node1-installer : ok=34   changed=1    unreachable=0    failed=0    skipped=25   rescued=0    ignored=0
ceph-saraut-rh-33mpct-node2 : ok=34   changed=1    unreachable=0    failed=0    skipped=25   rescued=0    ignored=0
ceph-saraut-rh-33mpct-node3 : ok=35   changed=2    unreachable=0    failed=0    skipped=25   rescued=0    ignored=0
ceph-saraut-rh-33mpct-node4 : ok=34   changed=1    unreachable=0    failed=0    skipped=25   rescued=0    ignored=0
ceph-saraut-rh-33mpct-node5 : ok=34   changed=1    unreachable=0    failed=0    skipped=25   rescued=0    ignored=0
localhost                  : ok=0    changed=0    unreachable=0    failed=0    skipped=4    rescued=0    ignored=0


TASK [Ensure reports directory exists on the Ansible controller] *******************************************************************************************
task path: /usr/share/cephadm-ansible/checks.yml:231
Wednesday 23 April 2025  07:01:06 +0000 (0:00:00.258)       0:01:05.453 *******
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1745391666.3140945-9644-87983463600153 `" && echo ansible-tmp-1745391666.3140945-9644-87983463600153="` echo /root/.ansible/tmp/ansible-tmp-1745391666.3140945-9644-87983463600153 `" ) && sleep 0'
Using module file /usr/lib/python3.9/site-packages/ansible/modules/file.py
<localhost> PUT /root/.ansible/tmp/ansible-local-8507rv3pezxt/tmpofkrbr_e TO /root/.ansible/tmp/ansible-tmp-1745391666.3140945-9644-87983463600153/AnsiballZ_file.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1745391666.3140945-9644-87983463600153/ /root/.ansible/tmp/ansible-tmp-1745391666.3140945-9644-87983463600153/AnsiballZ_file.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1745391666.3140945-9644-87983463600153/AnsiballZ_file.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1745391666.3140945-9644-87983463600153/ > /dev/null 2>&1 && sleep 0'
changed: [ceph-saraut-rh-33mpct-node3 -> localhost] => changed=true
  diff:
    after:
      path: ./reports
      state: directory
    before:
      path: ./reports
      state: absent
  gid: 0
  group: root
  invocation:
    module_args:
      _diff_peek: null
      _original_basename: null
      access_time: null
      access_time_format: '%Y%m%d%H%M.%S'
      attributes: null
      follow: true
      force: false
      group: null
      mode: '0755'
      modification_time: null
      modification_time_format: '%Y%m%d%H%M.%S'
      owner: null
      path: ./reports
      recurse: false
      selevel: null
      serole: null
      setype: null
      seuser: null
      src: null
      state: directory
      unsafe_writes: false
  mode: '0755'
  owner: root
  path: ./reports
  secontext: unconfined_u:object_r:usr_t:s0
  size: 6
  state: directory
  uid: 0


# sudo -l -U cephuser
Matching Defaults entries for cephuser on ceph-saraut-rh-33mpct-node1-installer:
    !visiblepw, always_set_home, match_group_by_gid, always_query_group_plugin, env_reset, env_keep="COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS",
    env_keep+="MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE", env_keep+="LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES",
    env_keep+="LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE", env_keep+="LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY",
    secure_path=/sbin\:/bin\:/usr/sbin\:/usr/bin

User cephuser may run the following commands on ceph-saraut-rh-33mpct-node1-installer:
    (ALL) NOPASSWD: ALL
Comment 5 errata-xmlrpc 2025-06-26 12:30:58 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:9775