Bug 1331735

Description of problem:
When using OSP 8 director to upgrade an OSP 7 installation that was originally deployed as 7.1 or 7.2, the converge step of the upgrade will fail with the following puppet error: 

puppet apply 0d5c39da-44bf-439c-9c01-9ccb3b6ea52a.pp
Could not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass
Could not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass
Error: Invalid resource type kmod::load at /var/lib/heat-config/heat-config-puppet/0d5c39da-44bf-439c-9c01-9ccb3b6ea52a.pp:19 on node overcloud-compute-0.localdomain
Wrapped exception:
Invalid resource type kmod::load
Error: Invalid resource type kmod::load at /var/lib/heat-config/heat-config-puppet/0d5c39da-44bf-439c-9c01-9ccb3b6ea52a.pp:19 on node overcloud-compute-0.localdomain

This occurs on controller, compute, and ceph nodes in my deployment.  

This is similar to these issues: 
https://ask.openstack.org/en/question/88671/heat-stack-failed-with-invalid-resource-type-kmodload/
https://bugzilla.redhat.com/show_bug.cgi?id=1310651

Basically the issue is that new puppet modules have been added to the image, but are not linked correctly in /etc/puppet.  You can see this by doing the following on one of the nodes: 

ls /usr/share/openstack-puppet/modules | while read line
do 
  if [ ! -L /etc/puppet/modules/$line ]; then 
    echo "$line not a link"
  fi
done

You'll see several modules not linked although kmod is the one being complained about.  I linked all remaining modules as follows: 

ls /usr/share/openstack-puppet/modules | while read line
do
  if [ ! -L /etc/puppet/modules/$line ]; then
    echo "Adding link /etc/puppet/modules/$line"
    ln -s /usr/share/openstack-puppet/modules/$line /etc/puppet/modules/$line
  fi
done

After this the converge deploy ran fine without the invalid kmod message.  



Version-Release number of selected component (if applicable):
RHEL OSP 8 / Director 8 GA 

How reproducible:
I've only tried once in my environment, but believe this will impact anyone originally starting with a deployment less than 7.3 

Steps to Reproduce:
1. Deploy RHEL OSP 7.1 or 7.2 
2. Update to latest 7.x packages
3. Follow documented process to upgrade from latest 7.x to 8.0 GA