Created attachment 1207247 [details]
versionlock list

My first thoughts, looking at the code:

  if $enable_ceph {
    $mon_initial_members = downcase(hiera('ceph_mon_initial_members'))
    if str2bool(hiera('ceph_ipv6', false)) {
      $mon_host = hiera('ceph_mon_host_v6')
    } else {
      $mon_host = hiera('ceph_mon_host')
    }
    class { '::ceph::profile::params': ## DUPLICATED #1
      mon_initial_members => $mon_initial_members,
      mon_host            => $mon_host,
    }
    include ::ceph::conf
    include ::ceph::profile::mon
  }

(...)
  
  if str2bool(hiera('enable_external_ceph', false)) {
    if str2bool(hiera('ceph_ipv6', false)) {
      $mon_host = hiera('ceph_mon_host_v6')
    } else { 
      $mon_host = hiera('ceph_mon_host')
    }
    class { '::ceph::profile::params': ## DUPLICATED #2
      mon_host            => $mon_host,
    }
    include ::ceph::conf
    include ::ceph::profile::client
  } 


It looks like both enable_ceph and enable_external_ceph are set to True, which leads to this duplicated resource.
I'm investigating in the env given to see is external_ceph is actually true.

Before updating the undercloud I run:
diff -Nar ~/pilot/templates/overcloud/ /usr/share/openstack-tripleo-heat-templates/|grep ^diff

diff -Nar /home/stack/pilot/templates/overcloud/puppet/hieradata/ceph.yaml /usr/share/openstack-tripleo-heat-templates/puppet/hieradata/ceph.yaml
diff -Nar /home/stack/pilot/templates/overcloud/puppet/manifests/overcloud_controller_pacemaker.pp /usr/share/openstack-tripleo-heat-templates/puppet/manifests/overcloud_controller_pacemaker.pp


Running diff on 2 overcloud_controller_pacemaker.pp I get:
557a558
>     include ::ceph::profile::rgw
999c1000,1001
<     enabled        => $non_pcmk_start,
---
>     # enabled        => $non_pcmk_start,
>     enabled        => false,
1875a1878,1884
>   if $ceph::profile::params::enable_rgw
>   {
>     exec { 'create_radosgw_keyring':
>       command => "/usr/bin/ceph auth get-or-create client.radosgw.gateway mon 'allow rwx' osd 'allow rwx' -o /etc/ceph/ceph.client.radosgw.gateway.keyring" ,
>       creates => "/etc/ceph/ceph.client.radosgw.gateway.keyring" ,
>     }
>   }


and I manually patch /usr/share/openstack-tripleo-heat-templates/puppet/manifests/overcloud_controller_pacemaker.pp with the diff after yum updating the undercloud.

as for /home/stack/pilot/templates/overcloud/puppet/hieradata/ceph.yaml 
I see that /usr/share/openstack-tripleo-heat-templates/puppet/hieradata/ceph.yaml  doesn't change when we run yum update on the undercloud, so I simply used the same copy of the file in update command.

Hi,

to further debug this it would be helpful to have:

 - /var/lib/heat-config/heat-config-puppet/ab53c213-94cc-4fd8-bf7f-258cf45b0bae.pp from overcloud-controller-2.localdomain;
 - the entire /etc/puppet/hieradata/ directory from the same host (small)
 - the /usr/share/openstack-puppet/modules/ directory could be useful as well (small when compressed)

It's could be that the class is still declared in hieradata, or that
as Emilien says that both parameters are set (enable_external_ceph and
ceph_mon_initial_members). With those files we should be able to find
the root cause.

Can we have a patch for it so we can drop it into JS-6.0?

Created attachment 1207650 [details]
/var/lib/heat-config/heat-config-puppet/ab53c213-94cc-4fd8-bf7f-258cf45b0bae.pp

Created attachment 1207652 [details]
hieradata dir from controller-2

Created attachment 1207660 [details]
/usr/share/openstack-puppet/modules/ from controller

Created attachment 1207661 [details]
/usr/share/openstack-puppet/modules/ from undercloud

Hi Sofer.
The requsted data is added.
Do you need anything else?
Thanks.

As per comment #3 from Emilien, I think they are deploying with settings for both managed *and* external ceph at the same time.

Could you collect from a controller node the output from:

$ sudo hiera enable_external_ceph

this is not supposed to return 'true', unless they are deploying with external ceph

[heat-admin@overcloud-controller-0 ~]$ sudo hiera enable_external_ceph
nil

Hi,

thanks Alexander for the information provided. I could reproduce
locally the error. The problem comes from the local modification of the
template done in
https://bugzilla.redhat.com/show_bug.cgi?id=1381629#c4 , most notably
the

   
    557a558
    >     include ::ceph::profile::rgw

This inclusion cause puppet to automatically include the
ceph::profile::params, hence the duplicate error:


    Debug: importing '/home/chem/Src/redhat/openstack-puppet-modules/ceph/manifests/profile/rgw.pp' in environment production
    Debug: Automatically imported ceph::profile::rgw from ceph/profile/rgw into production
    Warning: ModuleLoader: module 'ceph' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Debug: importing '/home/chem/Src/redhat/openstack-puppet-modules/ceph/manifests/profile/base.pp' in environment production
    Debug: Automatically imported ceph::profile::base from ceph/profile/base into production
    Debug: importing '/home/chem/Src/redhat/openstack-puppet-modules/ceph/manifests/profile/params.pp' in environment production
    Debug: Automatically imported ceph::profile::params from ceph/profile/params into production


I don't know exactly why the modification was done, but it cannot
work. You have to remove it from the list of your local modification
to have the minor upgrade get past this error.

I believe this is not a bug and occured as a result of wrongly patching /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml after runinng yum update.

Is this potentially related to how rgw is handled in this environment?

Hi, 

Alexander, Mike, you are both right.   This is not a bug, and it's related to the inclusion of the rados gateway in the manifest (include ::ceph::profile::rgw).  This local modification is blocking the upgrade.

So what is recommendation?
Replace offending line with ???
Fix handling duplicate ceph defintiion?

Update upgrade & update doc: remove offending lines, do update and upgrade, then update/upgrade rgw and other libraries manually/scripts and then bring lines back?

OSP10 will have new bits for RGW but nobody tried update or upgrade with them.

Sofer: Are you able to clearly articulate our recommendation to Dell and the answers to Arkady's questions in comment 18?

This is not a bug and was caused by a incorrect patch.  The correct procedure has been documented and verified by myself and Sasha.

Randy,
can we close this BZ?

So the error in this file turned out to have a duplication of the line 
"include ::ceph::profile::rgw"  once this was fixed, the error went away.  This bug can be closed.