Bug 1124573

Description of problem:

openshift.ks/openshift tries to download https://${broker_hostname}/rsync_id_rsa.pub in install_rsync_pub_key.  If there are multiple OpenShift broker hosts, they may have different rsync keys.  An OpenShift node host must have the public key of each OpenShift broker host so that all broker hosts can rsync gears between all node hosts.


How reproducible:

Readily enough.


Steps to Reproduce:

1. Use openshift.ks or openshift.sh to install an OpenShift Enterprise PaaS with at least two OpenShift broker hosts: broker01.hosts.example.com and broker02.hosts.example.com and at least one OpenShift node host: node01.hosts.example.com.

2. Run `ssh broker01.hosts.example.com ssh-keygen -lf /etc/openshift/rsync_id_rsa.pub`.

3. Run `ssh broker02.hosts.example.com ssh-keygen -lf /etc/openshift/rsync_id_rsa.pub`.

4. Run `ssh node01.hosts.example.com while read -r line \; do ssh-keygen -l -f /dev/stdin \<\<\<\$line \; done \< /root/.ssh/authorized_keys`.


Actual results:

Only one of the fingerprints from the output of Steps 2 and 3 appear in the output of Step 4; this tells us that the node host has only one of the broker hosts' public keys.


Expected results:

The output of Step 4 should include the fingerprints in the output of both Steps 2 and 3, which would indicate that the node host had both broker hosts' public keys.


Additional info:

An alternative to downloading each OpenShift broker host's public key on each node host would be to use a single private key on all broker hosts.  However, it is generally bad practice to share a private key, and distributing the same private key to each OpenShift broker host would probably be no easier than distributing each OpenShift broker host's public key to each OpenShift node host.

Currently, openshift.ks/openshift.sh has CONF_BROKER_HOSTNAME to specify one OpenShift broker host's hostname.  To resolve this bug, we will need to need to introduce an additional setting to specify the list of OpenShift broker hostnames.  This variable could be named CONF_BROKER_REPLICANTS (which may be misleading as "replicant" may suggest data replication), CONF_BROKER_HOSTNAMES (which is perhaps too easily confused with CONF_BROKER_HOSTNAME), or something else.  I can write the fix once we decide on a name for the setting.

Here is the relevant code in openshift.ks which fetches the rsync public key and will need to be modified to fix this bug:

https://github.com/openshift/openshift-extras/blob/6cf170ac916b832646b8edc41e195b5636045419/enterprise/install-scripts/openshift.ks#L2834-L2844