Bug 1469387
Summary: | fact gathering fails during upgrade from 3.4 to 3.5 on delete_empty_keys(facts['master']['admission_plugin_config']['BuildDefaults']['configuration']['env'])\r\nKeyError: 'env'\r\n" | ||||||
---|---|---|---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Javier Ramirez <javier.ramirez> | ||||
Component: | Cluster Version Operator | Assignee: | Scott Dodson <sdodson> | ||||
Status: | CLOSED ERRATA | QA Contact: | liujia <jiajliu> | ||||
Severity: | urgent | Docs Contact: | |||||
Priority: | urgent | ||||||
Version: | 3.5.0 | CC: | aos-bugs, ghuang, jokerman, mmccomas, nnosenzo, sdodson, wmeng | ||||
Target Milestone: | --- | ||||||
Target Release: | 3.5.z | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||
Doc Text: |
The configuration management for builddefaults attempts to remove environment variables that were previously defined but have since been removed from the configuration. In situations where no environment variables have been configured this was failing because the 'env' key did not exist. The process has now been updated to skip the cleanup when the env key does not exist.
|
Story Points: | --- | ||||
Clone Of: | |||||||
: | 1491718 1515457 1515458 1515459 (view as bug list) | Environment: | |||||
Last Closed: | 2018-01-09 20:18:58 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 1491718, 1515457, 1515458, 1515459 | ||||||
Attachments: |
|
Description
Javier Ramirez
2017-07-11 07:50:23 UTC
Created attachment 1296105 [details]
Ansible log
Customer used the following workaround: commenting out below line in mode python fact module (/usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py) fixed the issue and allowed us to upgrade: ------- 1422 #delete_empty_keys(facts['master']['admission_plugin_config']['BuildDefaults']['configuration']['env']) ------- Pull request open on https://github.com/openshift/openshift-ansible/pull/5375 Branch has been merged into * master * release-1.5 Still waiting on release-3.6 branch QE try to reproduced the bug with three ways, but did not get a failed upgrade. on atomic-openshift-utils-3.5.119-1.git.0.9e9bb4e.el7.noarch. 1, reproduce the bug as following steps: 1). Install ocp 3.4 with specified build config in inventory hosts file. <--snip--> openshift_buildoverrides_nodeselectors={'registry': 'enabled','router': 'enabled'} openshift_builddefaults_nodeselectors={'registry': 'enabled','router': 'enabled'} <--snip--> 2). Upgrade 3.4 to 3.5 with above inventory file. But I met another issue in step 1 that the above setting in step1 did not take effect for 3.4 installer. And I confirmed with related QE that "setting builddefaults and buildoverrides" is releasied from 3.5 and not support in 3.4. 2, Though it is not supported for upgrade that add the variables just during 3.4-3.5 upgrade but not during 3.4 installation. Try to reproduce the issue as following: 1). Install ocp 3.4 without any build config in inventory hosts file. 2). Upgrade 3.4 to 3.5 with following variables added in inventory hosts file. <--snip--> openshift_buildoverrides_nodeselectors={'registry': 'enabled','router': 'enabled'} openshift_builddefaults_nodeselectors={'registry': 'enabled','router': 'enabled'} Unfortunately, upgrade succeed and the variables did not take effect. 3, Try to reproduce the issue as following steps: 1) Install ocp 3.4 without any build config in inventory file. 2) Change master-config.yaml to add build config and then restart master service. admissionConfig: pluginConfig: BuildDefaults: configuration: apiVersion: v1 kind: BuildDefaultsConfig nodeSelector: registry: enabled router: enabled BuildOverrides: configuration: apiVersion: v1 kind: BuildOverridesConfig forcePull: true nodeSelector: registry: enabled router: enabled 3)Uprade 3.4 to 3.5 with following variables added in inventory hosts file. <--snip--> openshift_buildoverrides_nodeselectors={'registry': 'enabled','router': 'enabled'} openshift_builddefaults_nodeselectors={'registry': 'enabled','router': 'enabled'} Unfortunately, upgrade succeed and the variables keep original config. @Tim AFAIK, above variables are just for advanced install but not for upgrade. Would u mind give me more inspirations about how this issue re-produced? Because it can not be re-produced according to the config in attachment hosts file. So QE changed to use json format to config builddefaults, then it can be reproduced on version atomic-openshift-utils-3.5.120-1.git.0.c60f69a.el7.noarch. TASK [openshift_facts : Gather Cluster facts and set is_containerized if needed] ******************************************************************************************** fatal: [x.x.x.x]: FAILED! => { "changed": false, "failed": true, "module_stderr": "Shared connection to x.x.x.x closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_Hl3xm7/ansible_module_openshift_facts.py\", line 2489, in <module>\r\n main()\r\n File \"/tmp/ansible_Hl3xm7/ansible_module_openshift_facts.py\", line 2476, in main\r\n protected_facts_to_overwrite)\r\n File \"/tmp/ansible_Hl3xm7/ansible_module_openshift_facts.py\", line 1911, in __init__\r\n protected_facts_to_overwrite)\r\n File \"/tmp/ansible_Hl3xm7/ansible_module_openshift_facts.py\", line 1977, in generate_facts\r\n facts = set_builddefaults_facts(facts)\r\n File \"/tmp/ansible_Hl3xm7/ansible_module_openshift_facts.py\", line 1682, in set_builddefaults_facts\r\n delete_empty_keys(facts['master']['admission_plugin_config']['BuildDefaults']['configuration']['env'])\r\nKeyError: 'env'\r\n", "rc": 0 } MSG: MODULE FAILURE Version: atomic-openshift-utils-3.5.125-1.git.0.1c43b24.el7.noarch Steps: 1. Install ocp 3.4 with builddefaults setting in inventory hosts file. openshift_builddefaults_json={"BuildDefaults":{"configuration":{"apiVersion":"v1","nodeSelector":{"registry": "enabled"},"kind":"BuildDefaultsConfig"}}} 2. Upgrade 3.4 to 3.5 with above hosts file Upgrade still met the error. TASK [openshift_facts : Gather Cluster facts and set is_containerized if needed] ******************************************************************************************** fatal: [x.x.x.x]: FAILED! => { "changed": false, "failed": true, "module_stderr": "Shared connection to x.x.x.x closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_ZHVTSp/ansible_module_openshift_facts.py\", line 2491, in <module>\r\n main()\r\n File \"/tmp/ansible_ZHVTSp/ansible_module_openshift_facts.py\", line 2478, in main\r\n protected_facts_to_overwrite)\r\n File \"/tmp/ansible_ZHVTSp/ansible_module_openshift_facts.py\", line 1913, in __init__\r\n protected_facts_to_overwrite)\r\n File \"/tmp/ansible_ZHVTSp/ansible_module_openshift_facts.py\", line 1979, in generate_facts\r\n facts = set_builddefaults_facts(facts)\r\n File \"/tmp/ansible_ZHVTSp/ansible_module_openshift_facts.py\", line 1684, in set_builddefaults_facts\r\n delete_empty_keys(facts['master']['admission_plugin_config']['BuildDefaults']['configuration']['env'])\r\nKeyError: 'env'\r\n", "rc": 0 } Assign back, please let me know if my steps are not correct. Version: atomic-openshift-utils-3.5.128-1.git.0.f183c7b.el7.noarch Steps: 1. Install ocp 3.4 with builddefaults setting in inventory hosts file. openshift_builddefaults_json={"BuildDefaults":{"configuration":{"apiVersion":"v1","nodeSelector":{"registry": "enabled"},"kind":"BuildDefaultsConfig"}}} 2. Upgrade 3.4 to 3.5 with above hosts file Upgrade failed for the same error again. Any news on when this will be included in an errata ? I have tried the temporary workaround commented previously in comment #2: - Comment line delete_empty_keys(facts['master']['admission_plugin_config']['BuildDefaults']['configuration']['env']) in /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py But it didn't work, we're still getting: "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_6_ygCW/ansible_module_openshift_facts.py\", line 2491, in <module>\r\n main()\r\n File \"/tmp/ansible_6_ygCW/ansible_module_openshift_facts.py\", line 2478, in main\r\n protected_facts_to_overwrite)\r\n File \"/tmp/ansible_6_ygCW/ansible_module_openshift_facts.py\", line 1913, in __init__\r\n protected_facts_to_overwrite)\r\n File \"/tmp/ansible_6_ygCW/ansible_module_openshift_facts.py\", line 1979, in generate_facts\r\n facts = set_builddefaults_facts(facts)\r\n File \"/tmp/ansible_6_ygCW/ansible_module_openshift_facts.py\", line 1684, in set_builddefaults_facts\r\n delete_empty_keys(facts['master']['admission_plugin_config']['BuildDefaults']['configuration']['env'])\r\nKeyError: 'env'\r\n", Is there any other thing we can do to skip this and try to move on with the upgrade ? https://github.com/openshift/openshift-ansible/pull/6184 proposed fix against master @Scott About the three cloned bug, I tried to re-produce them but failed due to fresh install failed with openshift_builddefaults_json specified[1]. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1515746 Add cases. This fix is in openshift-ansible-3.7.10-1, openshift-ansible-3.6.173.0.81-1, openshift-ansible-3.5.146-1 all of which have shipped. It's also fixed in master as well. |