Description of problem:
Puppet does not deploy the same ceph.conf file to each of the nodes in a cluster. 
The Ceph "public_network"/Storage needs to be available to all Ceph using hosts. The Ceph "cluster_network"/Storage Cluster is needed only to the OSD nodes. 

However, if the Storage Cluster network is configure only to the OSD nodes then the ceph.conf file on the OSP Controller/Ceph monitor nodes does not not contain the "cluster_network" directive. 

Here is the ceph.conf file from an OSD followed by the ceph.conf from an OSP controller

OSD:
[global]
fsid = dae69645-9b4f-4a73-b93e-caafbf33a5dc
mon_initial_members = osp-cont-3, osp-cont-2, osp-cont-1
mon_host = 172.22.1.32, 172.22.1.30, 172.22.1.31
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true

public_network = 172.22.1.0/24
cluster_network = 172.22.2.0/24

[client.images]
keyring = /etc/ceph/ceph.client.images.keyring

[client.volumes]
keyring = /etc/ceph/ceph.client.volumes.keyring

Controller:
[global]
fsid = dae69645-9b4f-4a73-b93e-caafbf33a5dc
mon_initial_members = osp-cont-3, osp-cont-2, osp-cont-1
mon_host = 172.22.1.32, 172.22.1.30, 172.22.1.31
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true

public_network = 172.22.1.0/24

[client.images]
keyring = /etc/ceph/ceph.client.images.keyring

[client.volumes]
keyring = /etc/ceph/ceph.client.volumes.keyring



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: