Bug 1460421

Summary: Cannot upgrade OSP11 to OSP12: Overcloud upgrade failed due to : ERROR! no action detected in task
Product: Red Hat OpenStack Reporter: Artem Hrechanychenko <ahrechan>
Component: openstack-tripleo-heat-templatesAssignee: Jiri Stransky <jstransk>
Status: CLOSED ERRATA QA Contact: Gurenko Alex <agurenko>
Severity: urgent Docs Contact:
Priority: high    
Version: 12.0 (Pike)CC: ahrechan, jcoufal, jstransk, m.andre, mburns, mcornea, rhallise, rhel-osp-director-maint, sathlang, sclewis, tvignaud
Target Milestone: betaKeywords: Triaged
Target Release: 12.0 (Pike)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170706121722.el7ost openstack-tripleo-puppet-elements-7.0.0-0.20170703005458.0684eb4.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-13 21:31:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1399762    

Description Artem Hrechanychenko 2017-06-10 09:05:42 UTC
Description of problem:
017-06-10 08:48:25Z [overcloud-AllNodesDeploySteps-axkfftqzrjwt.ControllerUpgrade_Step0.0]: CREATE_FAILED  Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 4
2017-06-10 08:48:25Z [overcloud-AllNodesDeploySteps-axkfftqzrjwt.ControllerUpgrade_Step0]: CREATE_FAILED  Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 4
2017-06-10 08:48:26Z [overcloud-AllNodesDeploySteps-axkfftqzrjwt.ControllerUpgrade_Step0]: CREATE_FAILED  Error: resources.ControllerUpgrade_Step0.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 4
2017-06-10 08:48:26Z [overcloud-AllNodesDeploySteps-axkfftqzrjwt]: UPDATE_FAILED  Error: resources.ControllerUpgrade_Step0.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 4
2017-06-10 08:48:27Z [AllNodesDeploySteps]: UPDATE_FAILED  resources.AllNodesDeploySteps: Error: resources.ControllerUpgrade_Step0.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 4
2017-06-10 08:48:27Z [overcloud]: UPDATE_FAILED  resources.AllNodesDeploySteps: Error: resources.ControllerUpgrade_Step0.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 4

 Stack overcloud UPDATE_FAILED 

overcloud.AllNodesDeploySteps.ControllerUpgrade_Step0.0:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: 2a6b37b9-b991-422f-8bce-6c3bc640e8b1
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 4
  deploy_stdout: |

  deploy_stderr: |
    ERROR! no action detected in task. This often indicates a misspelled module name, or incorrect module path.
Heat Stack update failed.
Heat Stack update failed.

(undercloud) [stack@undercloud-0 ~]$ openstack stack failures list --long overcloud
overcloud.AllNodesDeploySteps.ControllerUpgrade_Step0.0:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: 2a6b37b9-b991-422f-8bce-6c3bc640e8b1
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 4
  deploy_stdout: |

  deploy_stderr: |
    ERROR! no action detected in task. This often indicates a misspelled module name, or incorrect module path.


Version-Release number of selected component (if applicable):
OSP12

How reproducible:


Steps to Reproduce:
1) Deploy OSP11 Undercloud & Overcloud
2) perform minor update of undercloud node
>sudo rhos-release 11 -r 7.4
>sudo systemctl stop 'openstack-*' 'neutron-*' httpd
>sudo yum update -y instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient
>openstack undercloud upgrade
>sudo reboot

3) upgrade osp11 undrcloud to osp12

>sudo rhos-release 12-director
>sudo systemctl stop openstack-*
>sudo systemctl stop neutron-*
>sudo systemctl stop httpd
>sudo yum -y update instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient
>openstack undercloud upgrade

#Overcloud upgrade 
source stackrc

export IMAGE_TAG=`bash /usr/bin/puddle-version http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/12.0-RHEL-7/latest_containers`

 sudo wget -O /home/stack/overcloud_containers.yaml http://file.rdu.redhat.com/~ohochman/containers/overcloud_containers.yaml
    sudo chown stack:stack /home/stack/overcloud_containers.yaml
    sudo sed -i 's:<IMAGE_TAG>:'$IMAGE_TAG':g' /home/stack/overcloud_containers.yaml
    sudo sed -i 's/--insecure-registry 192.168.24.3:8787/--insecure-registry docker-registry.engineering.redhat.com/' /etc/sysconfig/docker
    sudo service docker restart
    source /home/stack/stackrc && openstack overcloud container image upload --verbose --config-file /home/stack/overcloud_containers.yaml 
    
sudo wget -O /usr/share/openstack-tripleo-heat-templates/environments/docker-osp12.yaml http://file.rdu.redhat.com/~ohochman/containers/docker-osp12.yaml
    sudo chown stack:stack /usr/share/openstack-tripleo-heat-templates/environments/docker-osp12.yaml

    sudo sed -i 's:<IMAGE_TAG>:'$IMAGE_TAG':g' /usr/share/openstack-tripleo-heat-templates/environments/docker-osp12.yaml


# use master repos
cat > ~/containers-upgrade-repos.yaml <<EOEF
parameter_defaults:
  UpgradeInitCommand: |
    set -e
    curl -LO http://rhos-release.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm
    rpm -ivh rhos-release-latest.noarch.rpm || true
    rhos-release 12-director
    yum clean all
EOEF

export THT=/usr/share/openstack-tripleo-heat-templates

Apply workaround for https://bugzilla.redhat.com/show_bug.cgi?id=1456562
Apply workaround for https://bugzilla.redhat.com/show_bug.cgi?id=1458044 

# upgrade overcloud
openstack overcloud deploy --templates --libvirt-type kvm --ntp-server clock.redhat.com -e /home/stack/virt/network/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/virt/hostnames.yml -e /home/stack/virt/debug.yaml -e /home/stack/virt/nodes_data.yaml -e $THT/environments/low-memory-usage.yaml -e $THT/environments/docker.yaml -e $THT/environments/major-upgrade-composable-steps-docker.yaml -e ~/containers-upgrade-repos.yaml -e $THT/environments/docker-osp12.yaml --log-file overcloud_deployment_26.log




Actual results:
Overcloud upgrade was failed


Expected results:
Overcloud upgraded successfully

Comment 1 Red Hat Bugzilla Rules Engine 2017-06-10 09:05:47 UTC
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.

Comment 2 Artem Hrechanychenko 2017-06-19 13:49:38 UTC
didn't reproduced in last run

Comment 3 Omri Hochman 2017-06-19 15:55:18 UTC
Please check if reproduce on latest , if reproduces please check with dprince

Comment 4 Sofer Athlan-Guyot 2017-06-26 12:45:19 UTC
Hi,

we're moving this to triaged, as we're for reporter information.

Comment 5 Artem Hrechanychenko 2017-06-29 08:27:32 UTC
reproduced 
overcloud.AllNodesDeploySteps.ControllerUpgrade_Step0.0:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: 761dde55-fb3f-421e-8a42-a07c08cc3f3b
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 4
  deploy_stdout: |

  deploy_stderr: |
    ERROR! no action detected in task. This often indicates a misspelled module name, or incorrect module path.

Comment 6 Marius Cornea 2017-06-29 08:55:11 UTC
I suspect the issue is caused by missing installing ansible-pacemaker module during upgrade.

Comment 8 Jiri Stransky 2017-06-29 09:58:32 UTC
It should also be present in Pike images after we merge the patches, so we don't have to install this for P->Q again, and we may use it for minor updates too.

Comment 15 errata-xmlrpc 2017-12-13 21:31:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462