Bug 1504274
| Summary: | OSP11 -> OSP12 upgrade: major-upgrade-composable-steps-docker fails with An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: cannot import name shlex_quote | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Marius Cornea <mcornea> | ||||
| Component: | openstack-tripleo-heat-templates | Assignee: | Emilien Macchi <emacchi> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Marius Cornea <mcornea> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 12.0 (Pike) | CC: | aschultz, dbecker, emacchi, jschluet, mandreou, mbracho, mbultel, mburns, morazi, ohochman, rhel-osp-director-maint, sasha, sathlang | ||||
| Target Milestone: | rc | Keywords: | Triaged | ||||
| Target Release: | 12.0 (Pike) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-tripleo-heat-templates-7.0.3-11.el7ost | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2017-12-13 22:17:19 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Created attachment 1341212 [details]
debug info from mcornea env
Had a look at the environment that mcornea hit this BZ on.
I don't have any root cause and was unable to reproduce it by running the ansible playbook manually. Not sure who we need to get involved here ... is it ansible version/upgrade related?
Adding needinfo on DFG:DF TC - can we please get some help trying to triage this one? I remove the triaged keyword as we discussed today during the scrum triage call. For DFG:DF... is this something you are already seeing and or aware of - basically the ansible upgrade to 2.4 seems to causes this 'ImportError: cannot import name shlex_quote' (mcornea tried with version lock of ansible packages at some point to confirm). I will also ping sdoran now as he was suggested as a good person to ask. For now I leave DFG:DF off the whiteboards, please add as appropriate thanks very much This seems to be an ansible problem stemming from the upgrade tasks themselves. The failing task is a simple shell task so something is wrong in the package versions. Would need proper sosreports to investigate further. Most likely related to ansible 2.4.0 but this is not something the DFG:DF is currently familiar with. I tested the upgrades with ansible-2.4.1.0-1.el7ae.noarch.rpm and I wasn't able to reproduce the error reported initially in this ticket. From IRC, I asked about it. <dmsimard> mwhahaha: in 2.3 ansibles does from "ansible.compat.six.moves import shlex_quote", in 2.4 it's moved to "from ansible.module_utils.six.moves import shlex_quote" <dmsimard> mwhahaha: if we're using ansible to update itself, there's probably some weird things going on <@mwhahaha> dmsimard: ok if they ask again i'll try and remember that. Makes sense though if the running version was 2.3 and it gets updated underneath a run Given that it doesn't seem to be happening in ansible 2.4.1.0, sounds like it is addressed by that. thanks very much Alex. Given mcornea and your comments I think we can move this to ON_QA and in DFG:Upgrades - mcornea please move back if you disagree. (In reply to Marios Andreou from comment #7) > thanks very much Alex. Given mcornea and your comments I think we can move > this to ON_QA and in DFG:Upgrades - mcornea please move back if you disagree. I don't think this should be ON_QA. The test was done by manually installing ansible 2.4.1.0 before upgrade. In order to validate this we need the updated ansible package shipped in the repos deployed for testing. thanks mcornea ok then moving back to ASSIGNED at least so it doesn't look net new... I guess we can use this for tracking the availability of 2.4.1.0 in downstream builds. Going to reach out to reldel team today and we can discuss on our call later we discussed this on upgrades call last night - it may be that the new package still causes issues. The test that QE did was to update the package to 2.4.1 before running the upgrade workflow, rather than letting the update happen during the upgrade as usual. We can only confirm once we have the new package in puddle though and that already seems to be under way (email via jschlueter). If we continue to see the issue, another way we might go as discussed with mcornea briefly already is to use the UpgradeInit to run the ansible upgrade as one of the first tasks of the upgrade (after reposwitch, but before everything else) && mcornea was going to test like that too. ansible-2.4.1.0-1.el7ae.noarch.rpm is in the latest build but I was still able to reproduce the initial issue when running the upgrade cleanly(without installing the upgraded version before upgrade) https://review.openstack.org/#/c/517025/ updates the ansible package via yum before the upgrade starts to avoid ansible updating itself. openstack-tripleo-heat-templates-7.0.3-11.el7ost Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462 |
Description of problem: OSP11 -> OSP12 upgrade: major-upgrade-composable-steps-docker fails with An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: cannot import name shlex_quote: This is the failed upgrade tasks in the failures list output overcloud.AllNodesDeploySteps.ControllerUpgrade_Step3.1: resource_type: OS::Heat::SoftwareDeployment physical_resource_id: 77bef0af-a037-46a5-9f8a-b8d69dc07d5c status: CREATE_FAILED status_reason: | Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2 deploy_stdout: | PLAY [localhost] *************************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] TASK [Check if httpd is deployed] ********************************************** fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["systemctl", "is-enabled", "httpd"], "delta": "0:00:00.008017", "end": "2017-10-19 20:22:50.996250", "failed": true, "rc": 1, "start": "2017-10-19 20:22:50.988233", "stderr": "", "stderr_lines": [], "stdout": "disabled", "stdout_lines": ["disabled"]} ...ignoring TASK [Ensure mod_ssl package is installed] ************************************* changed: [localhost] TASK [Check if cinder_api is deployed] ***************************************** fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["systemctl", "is-enabled", "openstack-cinder-api"], "delta": "0:00:00.008898", "end": "2017-10-19 20:22:58.423753", "failed": true, "rc": 1, "start": "2017-10-19 20:22:58.414855", "stderr": "", "stderr_lines": [], "stdout": "disabled", "stdout_lines": ["disabled"]} ...ignoring TASK [Check if cinder_scheduler is deployed] *********************************** changed: [localhost] TASK [Check is heat_api is deployed] ******************************************* fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["systemctl", "is-enabled", "openstack-heat-api"], "delta": "0:00:00.005772", "end": "2017-10-19 20:22:58.858121", "failed": true, "rc": 1, "start": "2017-10-19 20:22:58.852349", "stderr": "", "stderr_lines": [], "stdout": "disabled", "stdout_lines": ["disabled"]} ...ignoring TASK [Check if heat_api_cfn is deployed] *************************************** fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["systemctl", "is-enabled", "openstack-heat-api-cfn"], "delta": "0:00:00.005196", "end": "2017-10-19 20:22:59.053675", "failed": true, "rc": 1, "start": "2017-10-19 20:22:59.048479", "stderr": "", "stderr_lines": [], "stdout": "disabled", "stdout_lines": ["disabled"]} ...ignoring TASK [Check if heat_api_cloudwatch is deployed] ******************************** fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["systemctl", "is-enabled", "openstack-heat-api-cloudwatch"], "delta": "0:00:00.005477", "end": "2017-10-19 20:22:59.257828", "failed": true, "rc": 1, "start": "2017-10-19 20:22:59.252351", "stderr": "", "stderr_lines": [], "stdout": "disabled", "stdout_lines": ["disabled"]} ...ignoring TASK [Check if neutron_server is deployed] ************************************* changed: [localhost] TASK [get bootstrap nodeid] **************************************************** changed: [localhost] TASK [set is_bootstrap_node fact] ********************************************** ok: [localhost] TASK [Stop pacemaker cluster] ************************************************** changed: [localhost] TASK [Check for mongodb service] *********************************************** ok: [localhost] TASK [Install docker packages on upgrade if missing] *************************** changed: [localhost] TASK [Upgrade os-net-config] *************************************************** changed: [localhost] TASK [take new os-net-config parameters into account now] ********************** changed: [localhost] TASK [Update all packages] ***************************************************** changed: [localhost] TASK [blank ipv6 rule before activating ipv6 firewall.] ************************ An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: cannot import name shlex_quote fatal: [localhost]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""} to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/0161d598-50c6-442f-9eeb-64a1ba779abf_playbook.retry PLAY RECAP ********************************************************************* localhost : ok=17 changed=14 unreachable=0 failed=1 deploy_stderr: |