Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1483121

Summary: OSP 9 -> 10 upgrade failure - ControllerPacemakerUpgradeDeployment_Step1 fails: Error: Unable to find constraint - 'Master'
Product: Red Hat OpenStack Reporter: Matt Flusche <mflusche>
Component: openstack-tripleo-heat-templatesAssignee: mathieu bultel <mbultel>
Status: CLOSED DUPLICATE QA Contact: Gurenko Alex <agurenko>
Severity: high Docs Contact:
Priority: urgent    
Version: 10.0 (Newton)CC: aschultz, augol, erik.young, mbultel, mburns, mcornea, rhel-osp-director-maint
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-21 13:06:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
bugfix patch none

Description Matt Flusche 2017-08-18 19:10:09 UTC
Description of problem:
During OSP 9 - 10 upgrade
ControllerPacemakerUpgradeDeployment_Step1 uses the following script to clear constraints for resources that were managed by pacemaker in osp 9.

/home/stack/templates/openstack-tripleo-heat-templates/extraconfig/tasks/major_upgrade_pacemaker_migrations.sh

        COL_CONSTRAINTS=$(pcs config show | sed -n '/^Colocation Constraints:$/,/^$/p' | grep -v "Colocation Constraints:" | egrep -v "ip-.*haproxy" | awk '{print $NF}' | cut -f2 -d: |cut -f1 -d\))
        for constraint in $COL_CONSTRAINTS; do
            log_debug "Deleting colocation constraint $constraint from CIB"
            pcs constraint remove "$constraint"
        done

It looks like the "pcs config show" output has changed and this method is no longer valid.

With an older version of pacemaker:

[root@mflusche-control000 ~]# rpm -q pacemaker
pacemaker-1.1.15-11.el7_3.2.x86_64
[root@mflusche-control000 ~]# pcs config show | sed -n '/^Colocation Constraints:$/,/^$/p' | grep -v "Colocation Constraints:" | egrep -v "ip-.*haproxy" | awk '{print $NF}' | cut -f2 -d: |cut -f1 -d\)
colocation-neutron-netns-cleanup-clone-neutron-ovs-cleanup-clone-INFINITY
colocation-openstack-aodh-listener-clone-openstack-aodh-evaluator-clone-INFINITY
colocation-openstack-ceilometer-api-clone-openstack-ceilometer-collector-clone-INFINITY
colocation-openstack-heat-engine-clone-openstack-heat-api-cloudwatch-clone-INFINITY
colocation-openstack-glance-api-clone-openstack-glance-registry-clone-INFINITY
colocation-openstack-cinder-scheduler-clone-openstack-cinder-api-clone-INFINITY
colocation-openstack-cinder-volume-openstack-cinder-scheduler-clone-INFINITY
colocation-openstack-nova-api-clone-openstack-nova-novncproxy-clone-INFINITY
colocation-openstack-nova-novncproxy-clone-openstack-nova-consoleauth-clone-INFINITY
colocation-neutron-metadata-agent-clone-neutron-l3-agent-clone-INFINITY
colocation-openstack-nova-scheduler-clone-openstack-nova-api-clone-INFINITY
colocation-openstack-aodh-evaluator-clone-delay-clone-INFINITY
colocation-delay-clone-openstack-ceilometer-api-clone-INFINITY
colocation-neutron-openvswitch-agent-clone-neutron-netns-cleanup-clone-INFINITY
colocation-openstack-heat-api-cloudwatch-clone-openstack-heat-api-cfn-clone-INFINITY
colocation-openstack-heat-api-cfn-clone-openstack-heat-api-clone-INFINITY
colocation-openstack-gnocchi-statsd-clone-openstack-gnocchi-metricd-clone-INFINITY
colocation-openstack-nova-conductor-clone-openstack-nova-scheduler-clone-INFINITY
colocation-neutron-l3-agent-clone-neutron-dhcp-agent-clone-INFINITY
colocation-neutron-dhcp-agent-clone-neutron-openvswitch-agent-clone-INFINITY
colocation-openstack-aodh-notifier-clone-openstack-aodh-evaluator-clone-INFINITY

And with the current version of pacemaker:

[root@mflusche-control000 ~]# rpm -q pacemaker
pacemaker-1.1.16-12.el7.x86_64

[root@mflusche-control000 ~]# pcs config show | sed -n '/^Colocation Constraints:$/,/^$/p' | grep -v "Colocation Constraints:" | egrep -v "ip-.*haproxy" | awk '{print $NF}' | cut -f2 -d: |cut -f1 -d\)
Master
Master
Master
Master
Master
Master
Master
Master
Master
Master
Master
Master
Master
Master
Master
Master
Master
INFINITY
INFINITY

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-5.2.0-25.el7ost.noarch
pacemaker-1.1.16-12.el7.x86_64


How reproducible:
100%

Steps to Reproduce:
1. Deploy and upgrade OSP 9 to current
2. Upgrade to osp 10 using current templates: openstack-tripleo-heat-templates-5.2.0-25.el7ost.noarch
3.

Actual results:


Expected results:


Additional info:

Comment 1 Matt Flusche 2017-08-18 21:54:49 UTC
It looks like it will fail in the next section of the script also.

        # Now we kill all the ordering constraints (except the haproxy/ip ones)
        ORD_CONSTRAINTS=$(pcs config show | sed -n '/^Ordering Constraints:/,/^Colocation Constraints:$/p' | grep -v "Ordering Constraints:"  | awk '{print $NF}' | cut -f2 -d: |cut -f1 -d\))
        for constraint in $ORD_CONSTRAINTS; do
            log_debug "Deleting ordering constraint $constraint from CIB"
            pcs constraint remove "$constraint"
        done


# pcs config show | sed -n '/^Ordering Constraints:/,/^Colocation Constraints:$/p' | grep -v "Ordering Constraints:"  |
 awk '{print $NF}' | cut -f2 -d: |cut -f1 -d\)
Optional
Optional
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
require-all=false
Mandatory
Mandatory
require-all=false
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory
Mandatory

Comment 2 Marius Cornea 2017-08-18 22:31:50 UTC
This looks similar to bug 1476711. Is it possible to check if the patch attached to bug 1476711 addresses the issue reported in this ticket? 

Thanks,
Marius

Comment 3 erik.young@lmco.com 2017-08-19 06:05:25 UTC
Yes. that is the bug. the fix does work.

We fixed it before seeing this note with:

COL_CONSTRAINTS=$(pcs constraint colocation show --full | awk -F: 'NR>1{gsub(/\)/, "") ; print $NF}')
ORD_CONSTRAINTS=$(pcs constraint order show --full | awk -F: 'NR>1{gsub(/\)/, "") ; print $NF}')

which are both more "correct" in this context, more concise, and work fine as far back as Juno.

-esy

Comment 4 erik.young@lmco.com 2017-08-19 06:14:03 UTC
patch is attached above.

-esy

Comment 5 erik.young@lmco.com 2017-08-19 06:15:11 UTC
Created attachment 1315626 [details]
bugfix patch

corrects the use of pcs config show to use pcs constraint instead with --full option.

Comment 6 Matt Flusche 2017-08-21 13:06:58 UTC
Yes, I'm going to close this as a dup of 1476711.

*** This bug has been marked as a duplicate of bug 1476711 ***