Bug 2174760

Summary: [Bluewash] cephadm-preflight playbook failing to upgrade the cephadm packages on nodes with message "IBM license agreement has not been accepted"
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Manisha Saini <msaini>
Component: BuildAssignee: Ken Dreyer (Red Hat) <kdreyer>
Status: NEW --- QA Contact: Manisha Saini <msaini>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 5.3CC: cephqe-warriors
Target Milestone: ---   
Target Release: 5.3z1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Manisha Saini 2023-03-02 12:08:26 UTC 
Description of problem:
====================

While performing upgrade from RHCS 5.3z1 --> IBM 5.3z1 , cephadm-preflight playbook is failing to upgrade the cephadm package on other nodes

---------
[cephuser@ceph-dsms-0l2782-node1-installer cephadm-ansible]$ sudo ansible-playbook -i hosts cephadm-preflight.yml --extra-vars "ceph_origin=custom upgrade_ceph_packages=true custom_repo_url=http://ceph-dsms-0l2782-node1-installer/repo/IBM-CEPH-5.3-20230228.ci.0"
[DEPRECATION WARNING]: [defaults]callback_whitelist option, normalizing names to new standard, use callbacks_enabled instead. This feature will be removed from ansible-core
 in version 2.15. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[WARNING]: Skipping callback plugin 'profile_tasks', unable to load

PLAY [localhost] ************************************************************************************************************************************************************

TASK [fail if insecure_registry is undefined] *******************************************************************************************************************************
skipping: [localhost]

PLAY [localhost] ************************************************************************************************************************************************************

TASK [fail when ceph_origin is custom with no repository defined] ***********************************************************************************************************
skipping: [localhost]

TASK [fail if baseurl is not defined for ceph_custom_repositories] **********************************************************************************************************
skipping: [localhost]

PLAY [all] ******************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************
ok: [ceph-dsms-0l2782-node4]
ok: [ceph-dsms-0l2782-node3]
ok: [ceph-dsms-0l2782-node2]
ok: [ceph-dsms-0l2782-node1-installer]

TASK [enable red hat storage tools repository] ******************************************************************************************************************************
skipping: [ceph-dsms-0l2782-node1-installer]
skipping: [ceph-dsms-0l2782-node3]
skipping: [ceph-dsms-0l2782-node2]
skipping: [ceph-dsms-0l2782-node4]

TASK [set_fact _ceph_repo] **************************************************************************************************************************************************
skipping: [ceph-dsms-0l2782-node1-installer]
skipping: [ceph-dsms-0l2782-node3]
skipping: [ceph-dsms-0l2782-node2]
skipping: [ceph-dsms-0l2782-node4]

TASK [configure ceph repository key] ****************************************************************************************************************************************
skipping: [ceph-dsms-0l2782-node1-installer]
skipping: [ceph-dsms-0l2782-node3]
skipping: [ceph-dsms-0l2782-node2]
skipping: [ceph-dsms-0l2782-node4]

TASK [configure ceph stable repository] *************************************************************************************************************************************
skipping: [ceph-dsms-0l2782-node1-installer]
skipping: [ceph-dsms-0l2782-node3]
skipping: [ceph-dsms-0l2782-node2]
skipping: [ceph-dsms-0l2782-node4]

TASK [fetch ceph red hat development repository] ****************************************************************************************************************************
skipping: [ceph-dsms-0l2782-node1-installer]
skipping: [ceph-dsms-0l2782-node3]
skipping: [ceph-dsms-0l2782-node2]
skipping: [ceph-dsms-0l2782-node4]

TASK [configure ceph red hat development repository] ************************************************************************************************************************
skipping: [ceph-dsms-0l2782-node1-installer]
skipping: [ceph-dsms-0l2782-node3]
skipping: [ceph-dsms-0l2782-node2]
skipping: [ceph-dsms-0l2782-node4]

TASK [set_fact ceph_custom_repositories] ************************************************************************************************************************************
ok: [ceph-dsms-0l2782-node1-installer]
ok: [ceph-dsms-0l2782-node3]
ok: [ceph-dsms-0l2782-node2]
ok: [ceph-dsms-0l2782-node4]

TASK [setup custom repositories] ********************************************************************************************************************************************
ok: [ceph-dsms-0l2782-node1-installer] => (item={'name': 'ceph_custom', 'description': 'Ceph custom repo', 'gpgcheck': 'no', 'state': 'present', 'baseurl': 'http://ceph-dsms-0l2782-node1-installer/repo/IBM-CEPH-5.3-20230228.ci.0', 'enabled': '1', 'file': 'ceph_custom', 'priority': '2'})
ok: [ceph-dsms-0l2782-node2] => (item={'name': 'ceph_custom', 'description': 'Ceph custom repo', 'gpgcheck': 'no', 'state': 'present', 'baseurl': 'http://ceph-dsms-0l2782-node1-installer/repo/IBM-CEPH-5.3-20230228.ci.0', 'enabled': '1', 'file': 'ceph_custom', 'priority': '2'})
ok: [ceph-dsms-0l2782-node4] => (item={'name': 'ceph_custom', 'description': 'Ceph custom repo', 'gpgcheck': 'no', 'state': 'present', 'baseurl': 'http://ceph-dsms-0l2782-node1-installer/repo/IBM-CEPH-5.3-20230228.ci.0', 'enabled': '1', 'file': 'ceph_custom', 'priority': '2'})
ok: [ceph-dsms-0l2782-node3] => (item={'name': 'ceph_custom', 'description': 'Ceph custom repo', 'gpgcheck': 'no', 'state': 'present', 'baseurl': 'http://ceph-dsms-0l2782-node1-installer/repo/IBM-CEPH-5.3-20230228.ci.0', 'enabled': '1', 'file': 'ceph_custom', 'priority': '2'})

TASK [install epel-release] *************************************************************************************************************************************************
skipping: [ceph-dsms-0l2782-node3]
skipping: [ceph-dsms-0l2782-node1-installer]
skipping: [ceph-dsms-0l2782-node2]
skipping: [ceph-dsms-0l2782-node4]

TASK [install ceph-common on rhel] ******************************************************************************************************************************************
ok: [ceph-dsms-0l2782-node3]
ok: [ceph-dsms-0l2782-node4]
ok: [ceph-dsms-0l2782-node2]
ok: [ceph-dsms-0l2782-node1-installer]

TASK [install prerequisites packages] ***************************************************************************************************************************************
ok: [ceph-dsms-0l2782-node1-installer]
FAILED - RETRYING: [ceph-dsms-0l2782-node3]: install prerequisites packages (3 retries left).
FAILED - RETRYING: [ceph-dsms-0l2782-node4]: install prerequisites packages (3 retries left).
FAILED - RETRYING: [ceph-dsms-0l2782-node2]: install prerequisites packages (3 retries left).
FAILED - RETRYING: [ceph-dsms-0l2782-node3]: install prerequisites packages (2 retries left).
FAILED - RETRYING: [ceph-dsms-0l2782-node4]: install prerequisites packages (2 retries left).
FAILED - RETRYING: [ceph-dsms-0l2782-node2]: install prerequisites packages (2 retries left).
FAILED - RETRYING: [ceph-dsms-0l2782-node3]: install prerequisites packages (1 retries left).
FAILED - RETRYING: [ceph-dsms-0l2782-node4]: install prerequisites packages (1 retries left).
FAILED - RETRYING: [ceph-dsms-0l2782-node2]: install prerequisites packages (1 retries left).
fatal: [ceph-dsms-0l2782-node3]: FAILED! => changed=false 
  attempts: 3
  failures:
  - - The IBM license agreement has not been accepted. Exiting.
    - Please install ibm-storage-ceph-license package first.
    - 'error: %pretrans(cephadm-2:16.2.10-140.el8cp.noarch) scriptlet failed, exit status 1'
  msg: Failed to install some of the specified packages
  rc: 1
  results: []
fatal: [ceph-dsms-0l2782-node4]: FAILED! => changed=false 
  attempts: 3
  failures:
  - - The IBM license agreement has not been accepted. Exiting.
    - Please install ibm-storage-ceph-license package first.
    - 'error: %pretrans(cephadm-2:16.2.10-140.el8cp.noarch) scriptlet failed, exit status 1'
  msg: Failed to install some of the specified packages
  rc: 1
  results: []
fatal: [ceph-dsms-0l2782-node2]: FAILED! => changed=false 
  attempts: 3
  failures:
  - - The IBM license agreement has not been accepted. Exiting.
    - Please install ibm-storage-ceph-license package first.
    - 'error: %pretrans(cephadm-2:16.2.10-140.el8cp.noarch) scriptlet failed, exit status 1'
  msg: Failed to install some of the specified packages
  rc: 1
  results: []

TASK [ensure chronyd is running] ********************************************************************************************************************************************
ok: [ceph-dsms-0l2782-node1-installer]

PLAY [localhost] ************************************************************************************************************************************************************

TASK [fail if insecure_registry is undefined] *******************************************************************************************************************************
skipping: [localhost]

PLAY [all] ******************************************************************************************************************************************************************

TASK [fail if insecure_registry is undefined] *******************************************************************************************************************************
skipping: [ceph-dsms-0l2782-node1-installer]

TASK [add registry as insecure registry in registries.conf] *****************************************************************************************************************
skipping: [ceph-dsms-0l2782-node1-installer]

PLAY RECAP ******************************************************************************************************************************************************************
ceph-dsms-0l2782-node1-installer : ok=6    changed=0    unreachable=0    failed=0    skipped=9    rescued=0    ignored=0   
ceph-dsms-0l2782-node2     : ok=4    changed=0    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0   
ceph-dsms-0l2782-node3     : ok=4    changed=0    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0   
ceph-dsms-0l2782-node4     : ok=4    changed=0    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0   
localhost                  : ok=0    changed=0    unreachable=0    failed=0    skipped=4    rescued=0    ignored=0   

--------------
All packages were upgraded except cephadm package
--------------

# rpm -qa | grep ceph
libcephfs2-16.2.10-140.el8cp.x86_64
python3-ceph-common-16.2.10-140.el8cp.x86_64
cephadm-16.2.10-138.el8cp.noarch
python3-cephfs-16.2.10-140.el8cp.x86_64
ceph-common-16.2.10-140.el8cp.x86_64
python3-ceph-argparse-16.2.10-140.el8cp.x86_64
ibm-storage-ceph-license-5.3-1.el8cp.noarch


Version-Release number of selected component (if applicable):

$ rpm -qa | grep cephadm
cephadm-ansible-1.14.0-1.el8cp.noarch
cephadm-16.2.10-140.el8cp.noarch


How reproducible:


Steps to Reproduce:
=====================
1.Install RHCS 5.3z1 cluster
2.Perform upgrade to IBM 5.3z1
  a) Upgrade the cephadm package on installer node ==> Successfully upgraded
  b) Run preflight playbook to upgrade packages on other nodes

Actual results:
=============
preflight playbook failed complaining to accept the Licence first


Expected results:
===========
Playbook should pass


Additional info:
Comment 1 Manisha Saini 2023-03-06 07:30:52 UTC 
Hi, 

I tried workaround of installing licencing package manually on all nodes and then run preflight playbook.In that case the playbook is passing.