Bug 1245176

Description of problem:
When failover happened in master HA env, only the 1st master works well.

Version-Release number of selected component (if applicable):
openshift-3.0.1.0-0.git.338.9dfce43.el7ose.x86_64
https://github.com/openshift/openshift-ansible (master branch)

How reproducible:
Always

Steps to Reproduce:
1. Install master HA env using the following inventory
# cat /etc/ansible/hosts | grep -v "^#"

[OSEv3:children]
masters
nodes
etcd

[OSEv3:vars]
ansible_ssh_user=root

deployment_type=enterprise

oreg_url=registry.access.redhat.com/openshift3/ose-${component}:${version}

openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/openshift/htpasswd'}]

openshift_master_cluster_password=openshift_cluster
openshift_master_cluster_vip=10.66.79.250
openshift_master_cluster_public_vip=10.66.79.250
openshift_master_cluster_hostname=master.cluster.local
openshift_master_cluster_public_hostname=master.cluster.local

[masters]
test[1:3].cluster.local

[etcd]
test[1:3].cluster.local

[nodes]
test[1:3].cluster.local openshift_scheduleable=False
test4.cluster.local openshift_node_labels="{'region': 'primary', 'zone': 'default'}"

2. After installation, openshift-master service is running in the 1st master.
# pcs status
Cluster name: openshift_master
Last updated: Tue Jul 21 19:12:45 2015
Last change: Tue Jul 21 16:52:18 2015
Stack: corosync
Current DC: test1.cluster.local (1) - partition with quorum
Version: 1.1.12-a14efad
3 Nodes configured
2 Resources configured


Online: [ test1.cluster.local test2.cluster.local test3.cluster.local ]

Full list of resources:

 Resource Group: openshift-master
     virtual-ip	(ocf::heartbeat:IPaddr2):	Started test1.cluster.local 
     master	(systemd:openshift-master):	Started test1.cluster.local 

PCSD Status:
  test1.cluster.local: Online
  test2.cluster.local: Online
  test3.cluster.local: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

3. Run `oc get nodes`, and install router successfully.
4. Stop the 1st node in master cluster.
# pcs cluster stop test1.cluster.local
test1.cluster.local: Stopping Cluster (pacemaker)...
test1.cluster.local: Stopping Cluster (corosync)...
5. openshift-master service is running on the 2nd node in master cluster.
# pcs status
Cluster name: openshift_master
Last updated: Tue Jul 21 19:32:14 2015
Last change: Tue Jul 21 16:52:18 2015
Stack: corosync
Current DC: test2.cluster.local (2) - partition with quorum
Version: 1.1.12-a14efad
3 Nodes configured
2 Resources configured


Online: [ test2.cluster.local test3.cluster.local ]
OFFLINE: [ test1.cluster.local ]

Full list of resources:

 Resource Group: openshift-master
     virtual-ip	(ocf::heartbeat:IPaddr2):	Started test2.cluster.local 
     master	(systemd:openshift-master):	Started test2.cluster.local 

PCSD Status:
  test1.cluster.local: Online
  test2.cluster.local: Online
  test3.cluster.local: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
7. Check master service log in the 2nd master, check log file for details
8. Run `oc get nodes`, it succeed.
9. Install docker-registry.

Actual results:
Wait for a long time, no docker-registry pod is created.
$ oc get rc
CONTROLLER          CONTAINER(S)   IMAGE(S)                                                             SELECTOR                                                                                REPLICAS
docker-registry-1   registry       registry.access.redhat.com/openshift3/ose-docker-registry:v3.0.1.0   deployment=docker-registry-1,deploymentconfig=docker-registry,docker-registry=default   0

Check master's log, get the following error.
Jul 21 19:26:58 test2 openshift-master: I0721 19:26:58.664278    7704 jwt.go:141] Signature error (key 0): crypto/rsa: verification error
Jul 21 19:26:58 test2 openshift-master: I0721 19:26:58.665260    7704 jwt.go:141] Signature error (key 0): crypto/rsa: verification error
Jul 21 19:26:58 test2 openshift-master: E0721 19:26:58.666060    7704 reflector.go:180] pkg/build/controller/factory/factory.go:69: Failed to list *api.Build: the server has asked for the client to provide credentials (get builds)
Jul 21 19:26:58 test2 openshift-master: E0721 19:26:58.667011    7704 reflector.go:180] /usr/lib/golang/src/runtime/asm_amd64.s:2232: Failed to list *api.ReplicationController: the server has asked for the client to provide credentials (get replicationControllers)
Jul 21 19:26:58 test2 openshift-master: I0721 19:26:58.685637    7704 jwt.go:141] Signature error (key 0): crypto/rsa: verification error
Jul 21 19:26:58 test2 openshift-master: I0721 19:26:58.687516    7704 factory.go:402] Failed to find any pods due to error the server has asked for the client to provide credentials (get pods)
Jul 21 19:26:58 test2 openshift-master: E0721 19:26:58.687591    7704 reflector.go:180] pkg/build/controller/factory/factory.go:105: Failed to list *api.Build: the server has asked for the client to provide credentials (get pods)

Force to make openshift-master failover to the 3rd master, the same behaviour.

When make opoenshift-master service get back to the 1st master, docker-registry pod is deployed successfully.

That means only the 1st master works well.

Expected results:
When failover happened, other master should work well.

Additional info: