This is a bit of a follow-on to https://bugzilla.redhat.com/show_bug.cgi?id=1599910

The more I think about this, the more I think that hieradata overrides are
a (painful) workaround for the fact that we're abusing composable networks
for an inappropriate use case.  This may be somewhat doable for compute
nodes, but it would seem to be almost completely impractical for any other
role, and there are legitimate reasons for putting other roles on "leaf"
networks:

1) We'd like to put a Glance API server in remote sites to act as a shared
   image cache for the compute nodes at that site.

2) Customers are very interested in having hyper-converged compute/Ceph
   nodes at remote sites.

3) For the non-remote spine & leaf case, it wouldn't be at all unreasonable
   for a customer to distribute control plane functions other than VIPs/
   HAProxy across their racks.

And I'm sure that there are others.

My gut tells me that the "right" way to do this would be to enable Neutron
routed networks for all of the overcloud networks (as we do for the control
plane network today).