Bug 2093414

Summary: Upgrade [OSP16.2 -> OSP17.1] ceph upgrade failing due to "No file was found when using first_found"
Product: Red Hat OpenStack Reporter: Juan Badia Payno <jbadiapa>
Component: tripleo-ansibleAssignee: Francesco Pantano <fpantano>
Status: CLOSED ERRATA QA Contact: Juan Badia Payno <jbadiapa>
Severity: high Docs Contact:
Priority: high    
Version: 17.0 (Wallaby)CC: alfrgarc, fpantano, johfulto, jpretori, jschluet, ltoscano, ramishra
Target Milestone: rcKeywords: Triaged
Target Release: 17.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tripleo-ansible-3.3.1-0.20220615001845.000acea.el8ost Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-08-16 01:11:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Juan Badia Payno 2022-06-03 16:06:36 UTC
After upgrading the undercloud from osp16.2 to osp17.0
Then the heat stack was updated.
 "openstack overcloud upgrade prepare" command
The the ceph upgrade from ceph4 to ceph5 was executed.
 "openstack overcloud external-upgrade --stack qe-Cloud-0 --tags ceph"

The following error was shown:

2022-05-28 11:47:08.729801 | 525400cf-ccf8-4dfd-e366-000000001670 |     TIMING | tripleo_ceph_run_ansible : run ceph-ansible | undercloud | 0:09:41.116734 | 561.98s                                                                         
2022-05-28 11:47:08.946081 | 525400cf-ccf8-4dfd-e366-000000001670 |     TIMING | tripleo_ceph_run_ansible : run ceph-ansible | undercloud | 0:09:41.333024 | 562.20s                                                                         
2022-05-28 11:47:09.076978 | 525400cf-ccf8-4dfd-e366-000000001671 |       TASK | search output of ceph-ansible run(s) non-zero return codes                                                                                                  
2022-05-28 11:47:09.944889 | 525400cf-ccf8-4dfd-e366-000000001671 |    SKIPPED | search output of ceph-ansible run(s) non-zero return codes | undercloud | result={                                                                          
    "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result",                                                                                                                              
    "changed": false                                                                                                                                                                                                                         
}                                                                                                                                                                                                                                            
2022-05-28 11:47:10.207290 | 525400cf-ccf8-4dfd-e366-000000001671 |     TIMING | tripleo_ceph_run_ansible : search output of ceph-ansible run(s) non-zero return codes | undercloud | 0:09:42.594229 | 1.13s                                 
2022-05-28 11:47:10.384937 | 525400cf-ccf8-4dfd-e366-000000001672 |       TASK | print ceph-ansible output in case of failure
2022-05-28 11:47:10.413530 | 525400cf-ccf8-4dfd-e366-000000001672 |    SKIPPED | print ceph-ansible output in case of failure | undercloud
2022-05-28 11:47:10.414043 | 525400cf-ccf8-4dfd-e366-000000001672 |     TIMING | tripleo_ceph_run_ansible : print ceph-ansible output in case of failure | undercloud | 0:09:42.800995 | 0.03s
2022-05-28 11:47:10.422072 | 525400cf-ccf8-4dfd-e366-0000000004e8 |       TASK | configure ceph clients
2022-05-28 11:47:10.439570 | 525400cf-ccf8-4dfd-e366-0000000004e8 |     TIMING | configure ceph clients | undercloud | 0:09:42.826515 | 0.02s
2022-05-28 11:47:10.508668 | 525400cf-ccf8-4dfd-e366-000000001922 |       TASK | Gather variables for each operating system
2022-05-28 11:47:10.537123 | 525400cf-ccf8-4dfd-e366-000000001922 |      FATAL | Gather variables for each operating system | undercloud | error={
    "msg": "No file was found when using first_found. Use errors='ignore' to allow this task to be skipped if no files are found"
}
2022-05-28 11:47:10.537568 | 525400cf-ccf8-4dfd-e366-000000001922 |     TIMING | tripleo_ceph_client : Gather variables for each operating system | undercloud | 0:09:42.924520 | 0.03s

NO MORE HOSTS LEFT ***************************************************************************************************************************************************************************************************************************

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
ceph-0                     : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2   
ceph-1                     : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2   
ceph-2                     : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2   
compute-0                  : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2   
compute-1                  : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2   
controller-0               : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2   
controller-1               : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2   
controller-2               : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=2   
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
undercloud                 : ok=75   changed=5    unreachable=0    failed=1    skipped=78   rescued=0    ignored=2   


2022-05-28 11:47:10.539809 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-05-28 11:47:10.539903 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 158        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-05-28 11:47:10.539990 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:09:42.926948 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-05-28 11:47:10.540084 |                                 UUID |       Info |       Host |   Task Name |   Run Time
2022-05-28 11:47:10.540163 | 525400cf-ccf8-4dfd-e366-000000001670 |    SUMMARY | undercloud | tripleo_ceph_run_ansible : run ceph-ansible | 562.20s
2022-05-28 11:47:10.540269 | 525400cf-ccf8-4dfd-e366-000000001426 |    SUMMARY | undercloud | tripleo_ceph_uuid : run nodes-uuid command | 3.31s
2022-05-28 11:47:10.540366 | 525400cf-ccf8-4dfd-e366-000000001662 |    SUMMARY | undercloud | tripleo_ceph_run_ansible : run create_ceph_ansible_remote_tmp command | 1.79s
2022-05-28 11:47:10.540454 | 525400cf-ccf8-4dfd-e366-000000001671 |    SUMMARY | undercloud | tripleo_ceph_run_ansible : search output of ceph-ansible run(s) non-zero return codes | 1.13s
2022-05-28 11:47:10.540536 | 525400cf-ccf8-4dfd-e366-000000000941 |    SUMMARY | undercloud | tripleo_ceph_work_dir : create ceph-ansible temp dirs | 0.84s
2022-05-28 11:47:10.540611 | 525400cf-ccf8-4dfd-e366-000000000eca |    SUMMARY | undercloud | tripleo_ceph_work_dir : create ceph-ansible temp dirs | 0.65s
2022-05-28 11:47:10.540696 | 525400cf-ccf8-4dfd-e366-000000000704 |    SUMMARY |  localhost | Gathering Facts | 0.60s
2022-05-28 11:47:10.540775 | 525400cf-ccf8-4dfd-e366-000000000957 |    SUMMARY | undercloud | tripleo_ceph_work_dir : generate ceph-ansible group vars all | 0.59s
2022-05-28 11:47:10.540863 | 525400cf-ccf8-4dfd-e366-000000000497 |    SUMMARY | undercloud | generate ceph-ansible group vars mgrs | 0.41s

Comment 5 John Fulton 2022-06-06 15:38:29 UTC
Please try again with the following:

parameter_defaults:
  CephConfigPath: "/etc/ceph"

We will need the documentation for the Ceph upgrade to include using the above.


Starting in 17 ceph client config is in /var/lib/tripleo-config/ceph but for older versions (and during adoption) it will be in the old location (/etc/ceph). 

  https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/cephadm/ceph-client.yaml#L58-L62

Comment 36 errata-xmlrpc 2023-08-16 01:11:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577