Bug 1298531 - Failed to upgrade ha environment if there are files in generated-configs
Failed to upgrade ha environment if there are files in generated-configs
Status: CLOSED CURRENTRELEASE
Product: OpenShift Container Platform
Classification: Red Hat
Component: Upgrade (Show other bugs)
3.1.0
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Andrew Butcher
Anping Li
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-14 06:26 EST by Anping Li
Modified: 2016-01-29 15:58 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-01-29 15:58:09 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Upgrade output (1.35 MB, text/plain)
2016-01-14 06:27 EST, Anping Li
no flags Details

  None (edit)
Description Anping Li 2016-01-14 06:26:19 EST
Description of problem:
If there are files under /etc/openshift/generated-configs in HA master environment, the upgrade will fail at "TASK: [openshift_master_certificates | file ]".


Version-Release number of selected component (if applicable):
atomic-openshift-utils-3.0.29-1

How reproducible:
always

Steps to Reproduce:
1. Set 3 masters (ose 3.0.2.0 ) Pacemaker HA environment.
2.  make sure there is /etc/openshift/generated-configs  on master1. 
This directory was generated by ansible.  In my HA Environment which installed by the previous ansible playbook, there is this directory on master1. but there aren't such directory on master2 , master3 and nodes.
3. ansible-playbook -i config/host3020clusterup /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_0_to_v3_1/upgrade.yml

Actual Result:

The upgrade failed. You can see message 'msg: Cannot link, file exists at destination'
For more information, please refer to the attached file

Expected Result:

Upgrade succeed


Addional:
There isn't such issue on single master environment.
If delete the /etc/openshift/generated-configs, the upgrade succeed
Comment 1 Anping Li 2016-01-14 06:27 EST
Created attachment 1114756 [details]
Upgrade output
Comment 2 Andrew Butcher 2016-01-14 09:42:07 EST
When certificates are created as part of the upgrade, we hard link those into the generated-configs dir for all masters. In this case, we created master.proxy-client.{crt,key} on the first master and those files must have already existed in /etc/openshift/generated-configs/<master-hostname>/.

Another possibility is that something else caused the installer to think certificates were missing but, since the log fails on master.proxy-client.key, I think we generated them during this run.

Was this upgrade ran against a fresh 3.0.2.0 install that hadn't been upgraded previously?
Comment 3 Anping Li 2016-01-19 21:06:06 EST
(In reply to Andrew Butcher from comment #2)
> When certificates are created as part of the upgrade, we hard link those
> into the generated-configs dir for all masters. In this case, we created
> master.proxy-client.{crt,key} on the first master and those files must have
> already existed in /etc/openshift/generated-configs/<master-hostname>/.
> 
> Another possibility is that something else caused the installer to think
> certificates were missing but, since the log fails on
> master.proxy-client.key, I think we generated them during this run.
> 
> Was this upgrade ran against a fresh 3.0.2.0 install that hadn't been
> upgraded previously?

Yes, It is a fresh 3.0.2.0 installed by prior ansible playbook. Which only generated-configs on the master master. 

If we installed 3.0.2.0 with latest ansible playbook. generated-configs  will be created on all masters/nodes.
Comment 4 Andrew Butcher 2016-01-20 14:47:41 EST
Proposed fix: https://github.com/openshift/openshift-ansible/pull/1245

Would you mind testing this from a checkout of the master branch of openshift-ansible?
Comment 5 Anping Li 2016-01-21 05:16:39 EST
Verified and pass

Note You need to log in before you can comment on or make changes to this bug.