Bug 1337335

Not sure how backward compatible hiera 2.x is but 3.0 update did break OpenStack a lot, see bug 1286826

This should be an RDO bug, then cloned into OSP.  moving back to RDO

Sample use case for [1] in TripleO [2]

1. https://docs.puppet.com/hiera/2.0/lookup_types.html#lookup-keys
2. https://review.openstack.org/#/c/318344/

Random facts collection:

* 2.x was short lived: March - June last year, then 3.0.1 was shipped
* 3.0.1 is known to break Packstack and TripleO,
* latest is 3.1.2 which was not tested, let's try that first and see if and what breaks in Packstack and TripleO

hiera-3.1.2 update is available for testing from a temporary repo:
http://cbs.centos.org/repos/cloud7-openstack-common-pending/x86_64/os/

Build in comment #5 worked fine for me on CentOS release 7.2.1511 and tripleo-heat-templates from master so I think it's good.

Thanks for testing!
hiera-3.1.2-1.el7 has been tagged into cloud7-openstack-common-testing for wider testing across all RDO release in all testing repos for all RDO releases ( http://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-*/ )

This is definitely breaking tripleo. hiera lookups fail when installing the undercloud:

2016-05-23 16:50:49 - Error: Could not find data item service_certificate in any Hiera data file and no default supplied at /etc/puppet/manifests/puppet-stack-config.pp:211 on node undercloud
2016-05-23 16:50:49 - Error: Could not find data item service_certificate in any Hiera data file and no default supplied at /etc/puppet/manifests/puppet-stack-config.pp:211 on node undercloud
+ rc=1
+ set -e
+ echo 'puppet apply exited with exit code 1'
puppet apply exited with exit code 1
+ '[' 1 '!=' 2 -a 1 '!=' 0 ']'
+ exit 1
[2016-05-23 16:50:49,523] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]
 
[2016-05-23 16:50:49,523] (os-refresh-config) [ERROR] Aborting...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1037, in install
    _run_orc(instack_env)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 915, in _run_orc
    _run_live_command(args, instack_env, 'os-refresh-config')
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 450, in _run_live_command
    raise RuntimeError('%s failed. See log for details.' % name)
RuntimeError: os-refresh-config failed. See log for details.
Command 'instack-install-undercloud' returned non-zero exit status 1

Downgrading the hiera package resolves the issue.

Giulio was able to track down the issue to the new hiera not liking nil values.

If I set undercloud_service_certificate = "''" in undercloud.conf, then the service_certificate key in hiera is populated with '' and the undercloud install succeeds.

We do have other cases in TripleO though where nil values end up in hiera. As such, I think it is still best to revert this, and make sure we have resolved those issues before updating.

hi John, thanks.

Still investigating on the root cause but the issue in comment #9 seems an instance of a more generic problem with the default values given to hiera() calls, specifically with 3.1 the following:

  puppet apply -e 'notice(hiera("service_certificate", "some_default"))'

does not seem to work as expected, giving:

  Error: Could not find data item service_certificate in any Hiera data file and no default supplied at line 1 on node instack

while it is instead supposed to return "some_default".

So is the recommended method to proceed with installation now, to set undercloud_service_certificate = "''" or to downgrade hiera?

I think downgrade is the best short-term option pending further investigation?

I posted this which (if I have my exclude syntax correct?) should exclude just the hiera-3* version and fall back to the old version for tripleo-ci:

https://review.openstack.org/#/c/320127/

http://cbs.centos.org/koji/buildinfo?buildID=11136 hiera 1.3.4 with bumped Epoch:1 was pushed to the common-testing.

Available in testing repo used by CI jobs:
http://buildlogs.centos.org/centos/7/cloud/x86_64/openstack-mitaka/common/hiera-1.3.4-5.el7.noarch.rpm

@EmilienM - original issue is back on hold, until nil values issue is resolved in TripleO.

It seems like we're not alone,

See:
https://projects.puppetlabs.com/issues/20923
https://github.com/camptocamp/puppet-hiera-undef

I'm wondering if a recent version of Puppet would help us.
I'll take some time to test it.

I guess this can be closed now. Not an issue for me any longer.