Bug 1298531

Summary: Failed to upgrade ha environment if there are files in generated-configs
Product: OpenShift Container Platform Reporter: Anping Li <anli>
Component: Cluster Version OperatorAssignee: Andrew Butcher <abutcher>
Status: CLOSED CURRENTRELEASE QA Contact: Anping Li <anli>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.1.0CC: abutcher, aos-bugs, bleanhar, jokerman, mmccomas
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-29 20:58:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Upgrade output none

Description Anping Li 2016-01-14 11:26:19 UTC
Description of problem:
If there are files under /etc/openshift/generated-configs in HA master environment, the upgrade will fail at "TASK: [openshift_master_certificates | file ]".


Version-Release number of selected component (if applicable):
atomic-openshift-utils-3.0.29-1

How reproducible:
always

Steps to Reproduce:
1. Set 3 masters (ose 3.0.2.0 ) Pacemaker HA environment.
2.  make sure there is /etc/openshift/generated-configs  on master1. 
This directory was generated by ansible.  In my HA Environment which installed by the previous ansible playbook, there is this directory on master1. but there aren't such directory on master2 , master3 and nodes.
3. ansible-playbook -i config/host3020clusterup /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_0_to_v3_1/upgrade.yml

Actual Result:

The upgrade failed. You can see message 'msg: Cannot link, file exists at destination'
For more information, please refer to the attached file

Expected Result:

Upgrade succeed


Addional:
There isn't such issue on single master environment.
If delete the /etc/openshift/generated-configs, the upgrade succeed

Comment 1 Anping Li 2016-01-14 11:27:22 UTC
Created attachment 1114756 [details]
Upgrade output

Comment 2 Andrew Butcher 2016-01-14 14:42:07 UTC
When certificates are created as part of the upgrade, we hard link those into the generated-configs dir for all masters. In this case, we created master.proxy-client.{crt,key} on the first master and those files must have already existed in /etc/openshift/generated-configs/<master-hostname>/.

Another possibility is that something else caused the installer to think certificates were missing but, since the log fails on master.proxy-client.key, I think we generated them during this run.

Was this upgrade ran against a fresh 3.0.2.0 install that hadn't been upgraded previously?

Comment 3 Anping Li 2016-01-20 02:06:06 UTC
(In reply to Andrew Butcher from comment #2)
> When certificates are created as part of the upgrade, we hard link those
> into the generated-configs dir for all masters. In this case, we created
> master.proxy-client.{crt,key} on the first master and those files must have
> already existed in /etc/openshift/generated-configs/<master-hostname>/.
> 
> Another possibility is that something else caused the installer to think
> certificates were missing but, since the log fails on
> master.proxy-client.key, I think we generated them during this run.
> 
> Was this upgrade ran against a fresh 3.0.2.0 install that hadn't been
> upgraded previously?

Yes, It is a fresh 3.0.2.0 installed by prior ansible playbook. Which only generated-configs on the master master. 

If we installed 3.0.2.0 with latest ansible playbook. generated-configs  will be created on all masters/nodes.

Comment 4 Andrew Butcher 2016-01-20 19:47:41 UTC
Proposed fix: https://github.com/openshift/openshift-ansible/pull/1245

Would you mind testing this from a checkout of the master branch of openshift-ansible?

Comment 5 Anping Li 2016-01-21 10:16:39 UTC
Verified and pass