Description of problem: RHOSP overcloud deploy failing after patching the director Version-Release number of selected component (if applicable): RedHat OpenStack Director 10 How reproducible: Customer Environment Steps to Reproduce: 1. Built an undercloud with 7.2 and patched up to 7.3: - deployed a simple overcloud with just a controller and compute and no custom scripts - resulted in a successful deployment. $ openstack overcloud deploy --templates --stack ocd98 --control-scale 1 --compute-scale 1 --control-flavor control --compute-flavor compute --ntp-server 10.224.0.5 --libvirt-type qemu --debug --log-file overcloud_deployment.log - deployed our full overcloud with customer scripts [3] - resulted in a successful deployment. 2. Updated the undercloud to 7.4 - deployed a simple overcloud with just a controller and compute and no customer scripts [3] - resulted in a failed deployment. 3. Customer deployment script: openstack overcloud deploy $DRYP --templates \ --stack $STACK_PREFIX --ntp-server $OSPD_PXE \ --control-flavor control --control-scale $CTL_SCALE \ --compute-flavor compute --compute-scale $CMP_SCALE \ --ceph-storage-scale $STO_SCALE \ --neutron-network-type vxlan \ --neutron-tunnel-types vxlan \ --debug --log-file overcloud_deployment.log \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/net-bond-with-vlans.yaml \ -e /home/stack/templates/firstboot-environment.yaml \ -e /home/stack/templates/pre_config.yaml \ -e /home/stack/templates/timezone.yaml \ -e /home/stack/templates/network-environment.yaml \ -e /home/stack/templates/puppet-ceph-external.yaml \ -e /home/stack/templates/inject-trust-anchor-hiera.yaml \ -e /home/stack/templates/tls-endpoints-public-dns.yaml \ -e /home/stack/templates/enable-tls.yaml \ -e /home/stack/templates/munchausen.yaml \ -e /home/stack/templates/post_config.yaml 4. The customer has to configure AD in a post script that is copied over to controller-0 from the post section of our script on the director Call from the director: ssh heat-admin@$STACK_PREFIX-controller-0 sudo /tmp/keystone/create_AD_domain.sh Script run on controller: #!/bin/bash - # Give this script it's actual name for log spelunking script_name=create_keystone_domain # Configure OSPD Host/User # -jrm OSPD_Host="10.224.0.5" OSPD_User="stack" OSPD_Stackname=$(ssh $OSPD_User@$OSPD_Host "awk -F= '/^export STACK_PREFIX=/{print \$2}' ~/deploy.sh" | sed 's/"//g') # Get Overcloud RC file RCNAME=${OSPD_Stackname}rc_v3 scp $OSPD_User@$OSPD_Host:~/$RCNAME /tmp/keystone/overcloudrc source /tmp/keystone/overcloudrc logger "$script_name - Create Keystone domain." DOMAIN_NAME="AD" # if openstack domain list | grep -q $DOMAIN_NAME'.*True' ; then if openstack domain show $DOMAIN_NAME | grep -q $DOMAIN_NAME ; then logger "$script_name - Domain $DOMAIN_NAME already exists -- skipping creation" else logger "$script_name - Missing $DOMAIN_NAME Domain -- creating now" openstack domain create $DOMAIN_NAME logger "$script_name - Created AD domain." # Assign admin role to admin user for new domain export DOMAIN_ID=$(openstack domain show -f shell $DOMAIN_NAME | awk -F= '/id/{print $2}' | sed 's/"//g') export DOMAIN_ADMIN=$(openstack user list -f csv --domain default | awk -F, '/admin/{print $1}' | sed 's/"//g') export DOMAIN_ADMIN_ROLE=$(openstack role list -f csv | awk -F, '/admin/{print $1}' | sed 's/"//g') logger "$script_name - Executing: openstack role add --domain $DOMAIN_ID --user $DOMAIN_ADMIN $DOMAIN_ADMIN_ROLE" openstack role add --domain $DOMAIN_ID --user $DOMAIN_ADMIN $DOMAIN_ADMIN_ROLE openstack role add --domain default --user admin admin fi ### eof ### The related keystone.AD.conf ~ [ldap] url = ldaps://<elided> user = <elided> password = <elided> suffix = <elided> group_tree_dn = <elided> user_tree_dn = <elided> query_scope=sub page_size=5 debug_level=4095 user_filter=(<elided>) group_objectclass=group group_filter=(<elided>) group_name_attribute = name user_objectclass=user user_id_attribute=sAMAccountName user_name_attribute=sAMAccountName user_pass_attribute= user_enabled_attribute=userAccountControl user_enabled_mask = 2 user_enabled_default = 512 user_attribute_ignore = password,tenant_id,tenants user_allow_create=false user_allow_update=false user_allow_delete=false role_allow_create=false role_allow_update=false role_allow_delete=false group_objectclass=group group_allow_create=false group_allow_update=false group_allow_delete=false tls_cacertfile=/etc/pki/ca-trust/source/anchors/ca_crt.pem use_tls=false [identity] driver = keystone.identity.backends.ldap.Identity ~ Actual results: Deployment has failed with SwiftRingupdate issue ~~~ RESP BODY: {"events": [{"resource_name": "SwiftRingUpdate" "event_time": "2017-08-31T16:09:42Z" "links": [{"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerSwiftRingUpdate-d6xnuqvsztx2/0f1da701-d8e9-4071-9863-4ffc32bc0ce0/resources/SwiftRingUpdate/events/61172b51-2798-4a91-b201-2785fdc1b5f4" "rel": "self"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerSwiftRingUpdate-d6xnuqvsztx2/0f1da701-d8e9-4071-9863-4ffc32bc0ce0/resources/SwiftRingUpdate" "rel": "resource"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerSwiftRingUpdate-d6xnuqvsztx2/0f1da701-d8e9-4071-9863-4ffc32bc0ce0" "rel": "stack"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98/d37da06f-977d-4bf5-a924-1a1a4213a626" "rel": "root_stack"}] "logical_resource_id": "SwiftRingUpdate" "resource_status": "CREATE_FAILED" "resource_status_reason": "Error: resources.SwiftRingUpdate.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1" "physical_resource_id": "2a1cd88e-4b40-4c3b-9a88-32fefc0084dc" "id": "61172b51-2798-4a91-b201-2785fdc1b5f4"} {"resource_name": "ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerSwiftRingUpdate-d6xnuqvsztx2" "event_time": "2017-08-31T16:09:42Z" "links": [{"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerSwiftRingUpdate-d6xnuqvsztx2/0f1da701-d8e9-4071-9863-4ffc32bc0ce0/resources/ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerSwiftRingUpdate-d6xnuqvsztx2/events/21132a7a-634c-44b1-9722-64a5e33cf03e" "rel": "self"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerSwiftRingUpdate-d6xnuqvsztx2/0f1da701-d8e9-4071-9863-4ffc32bc0ce0/resources/ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerSwiftRingUpdate-d6xnuqvsztx2" "rel": "resource"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerSwiftRingUpdate-d6xnuqvsztx2/0f1da701-d8e9-4071-9863-4ffc32bc0ce0" "rel": "stack"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98/d37da06f-977d-4bf5-a924-1a1a4213a626" "rel": "root_stack"}] "logical_resource_id": "ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerSwiftRingUpdate-d6xnuqvsztx2" "resource_status": "CREATE_FAILED" "resource_status_reason": "Resource CREATE failed: Error: resources.SwiftRingUpdate.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1" "physical_resource_id": "0f1da701-d8e9-4071-9863-4ffc32bc0ce0" "id": "21132a7a-634c-44b1-9722-64a5e33cf03e"} {"resource_name": "ControllerSwiftRingUpdate" "event_time": "2017-08-31T16:09:44Z" "links": [{"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu/c4a3f74d-f101-4f85-b0a3-9a4f32b3ab52/resources/ControllerSwiftRingUpdate/events/6dfffedf-872e-42d7-b5eb-61d8bdfb801b" "rel": "self"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu/c4a3f74d-f101-4f85-b0a3-9a4f32b3ab52/resources/ControllerSwiftRingUpdate" "rel": "resource"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu/c4a3f74d-f101-4f85-b0a3-9a4f32b3ab52" "rel": "stack"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98/d37da06f-977d-4bf5-a924-1a1a4213a626" "rel": "root_stack"}] "logical_resource_id": "ControllerSwiftRingUpdate" "resource_status": "CREATE_FAILED" "resource_status_reason": "Error: resources.ControllerSwiftRingUpdate.resources.SwiftRingUpdate.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1" "physical_resource_id": "0f1da701-d8e9-4071-9863-4ffc32bc0ce0" "id": "6dfffedf-872e-42d7-b5eb-61d8bdfb801b"} {"resource_name": "ControllerPostPuppet" "event_time": "2017-08-31T16:09:44Z" "links": [{"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu/c4a3f74d-f101-4f85-b0a3-9a4f32b3ab52/resources/ControllerPostPuppet/events/25dcb77c-2766-4f95-afbb-e52f376a07b2" "rel": "self"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu/c4a3f74d-f101-4f85-b0a3-9a4f32b3ab52/resources/ControllerPostPuppet" "rel": "resource"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu/c4a3f74d-f101-4f85-b0a3-9a4f32b3ab52" "rel": "stack"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98/d37da06f-977d-4bf5-a924-1a1a4213a626" "rel": "root_stack"}] "logical_resource_id": "ControllerPostPuppet" "resource_status": "CREATE_FAILED" "resource_status_reason": "CREATE aborted" "physical_resource_id": "96796187-04cf-4e8f-9397-98ef45a50855" "id": "25dcb77c-2766-4f95-afbb-e52f376a07b2"} {"resource_name": "ControllerPostPuppetMaintenanceModeDeployment" "event_time": "2017-08-31T16:09:44Z" "links": [{"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerPostPuppet-giwwby2cnqhx/96796187-04cf-4e8f-9397-98ef45a50855/resources/ControllerPostPuppetMaintenanceModeDeployment/events/ac328343-4181-4790-8d93-04f3549b883b" "rel": "self"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerPostPuppet-giwwby2cnqhx/96796187-04cf-4e8f-9397-98ef45a50855/resources/ControllerPostPuppetMaintenanceModeDeployment" "rel": "resource"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerPostPuppet-giwwby2cnqhx/96796187-04cf-4e8f-9397-98ef45a50855" "rel": "stack"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98/d37da06f-977d-4bf5-a924-1a1a4213a626" "rel": "root_stack"}] "logical_resource_id": "ControllerPostPuppetMaintenanceModeDeployment" "resource_status": "CREATE_FAILED" "resource_status_reason": "CREATE aborted" "physical_resource_id": "0983534b-5d8c-4dcf-87cf-32839e4841bc" "id": "ac328343-4181-4790-8d93-04f3549b883b"} {"resource_name": "ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerPostPuppet-giwwby2cnqhx" "event_time": "2017-08-31T16:09:44Z" "links": [{"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerPostPuppet-giwwby2cnqhx/96796187-04cf-4e8f-9397-98ef45a50855/resources/ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerPostPuppet-giwwby2cnqhx/events/e3f8f55b-77a8-4f6c-b426-e7f5af3f6283" "rel": "self"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerPostPuppet-giwwby2cnqhx/96796187-04cf-4e8f-9397-98ef45a50855/resources/ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerPostPuppet-giwwby2cnqhx" "rel": "resource"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerPostPuppet-giwwby2cnqhx/96796187-04cf-4e8f-9397-98ef45a50855" "rel": "stack"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98/d37da06f-977d-4bf5-a924-1a1a4213a626" "rel": "root_stack"}] "logical_resource_id": "ocd98-AllNodesDeploySteps-hhxvxdqccqeu-ControllerPostPuppet-giwwby2cnqhx" "resource_status": "CREATE_FAILED" "resource_status_reason": "Resource CREATE failed: Operation cancelled" "physical_resource_id": "96796187-04cf-4e8f-9397-98ef45a50855" "id": "e3f8f55b-77a8-4f6c-b426-e7f5af3f6283"} {"resource_name": "ocd98-AllNodesDeploySteps-hhxvxdqccqeu" "event_time": "2017-08-31T16:09:44Z" "links": [{"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu/c4a3f74d-f101-4f85-b0a3-9a4f32b3ab52/resources/ocd98-AllNodesDeploySteps-hhxvxdqccqeu/events/ab3b919f-9865-4bc9-8d08-0a1f2eb7998b" "rel": "self"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu/c4a3f74d-f101-4f85-b0a3-9a4f32b3ab52/resources/ocd98-AllNodesDeploySteps-hhxvxdqccqeu" "rel": "resource"} {"href": "http://10.224.0.5:8004/v1/5cb69c8745494a3d9fb7b9b9202e5858/stacks/ocd98-AllNodesDeploySteps-hhxvxdqccqeu/c4a3f74d-f101-4f85-b0a3-9a4f32b3ab52" "rel": "stack"} {"href": "htt: ~~~ Expected results: The deployment should complete successfully after patching the UC from 7.3 to 7.4 Additional info: 1. SOS report details on below private comment. 2. I found the similar issue which has reported on below BZ. Please help us to provide the hotfix if the issue is indicating to swiftringupdate issue. https://bugzilla.redhat.com/show_bug.cgi?id=1468030 https://bugzilla.redhat.com/show_bug.cgi?id=1459919 Please let me know if you need any further information on it. Regards, pradipta
Most likely a duplicate of Bug 1459919 as that has yet to be released.
I agree with Alex - it's a duplicate of 1459919. The return code of the Swift ring rebalance is 1, and that results in a failure. The fix is in the updated puppet-swift-9.5.0-3.el7ost package. Closing this one therefore and marking it as duplicate. *** This bug has been marked as a duplicate of bug 1459919 ***