Description of problem:

Assume a geo-replication topography where two different hosts with a volume both replicate to a central remote host having two volumes, and the central host then re-mirrors origin node data to the opposing node on a seperate volume in a chained fashion.

Like so, (capital letter is a host, lowercase is a volume)
A(a) -> C(a) -> B(a)
B(b) -> C(b) -> A(b)

So far this seems to work, expect in the case of initial creation:  

- The push-pem option of create replaces common_secret.pem.pub on the target host.

- When that target host is also a replication master, this replaces the master common_secret.pem.pub on this host.

- Create commands after this event on the second host will then push the common_secret.pem.pub containing the masters' masters' keys (eg, keys from step 1).

- This breaks start since the pubkeys are wrong.

- Start reports a success, even though the session doesn't start and the only thing in the log is a permission denied error from using the wrong key.


Version-Release number of selected component (if applicable):
3.5.2

How reproducible:
Always

Steps to Reproduce:
1. Create chained replication as described above
2. Note that common_secret.pem.pub has the "wrong" keys for a replication master on some subset of hosts.
3. Note that start doesn't start

Actual results:
All session creation and starting works as expected

Expected results:
Subset of nodes need their keys recreated again and pushed in order to function correctly.

Additional info:

It looks like common_secret.pem.pub is a shortcut for only having to copy around one file instead of two.  Probably the easiest fix would be to change the name of this file to contain a host string, and modify the command that use it to require the file-name instead of assuming it.

For future folks: The workaround is to run "gluster system:: execute gsec_create" again which will regenerate common_secret.pem.pub for the host without changing the keys.  You can then re-run the "create push-pem" step to recoy keys after which, a start will work correctly.  It seems like you could also use OOB management (ansible, puppet, etc..) to copy the keys around and skip the push-pem business.