Bug 1314840 - Upgrade from OSP 7.0.3 to 7.0.4 breaks if there is no partitions for swift to rebalance. [NEEDINFO]
Upgrade from OSP 7.0.3 to 7.0.4 breaks if there is no partitions for swift to...
Status: CLOSED DUPLICATE of bug 1321088
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity medium
: async
: 7.0 (Kilo)
Assigned To: Angus Thomas
Arik Chernetsky
: Unconfirmed, ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-04 11:24 EST by Jeremy
Modified: 2016-05-03 12:49 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-03 12:49:30 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
jmelvin: needinfo? (athomas)
clincoln: needinfo? (athomas)


Attachments (Terms of Use)

  None (edit)
Description Jeremy 2016-03-04 11:24:35 EST
Description of problem:

During an update test,  the process was interrupted and marked in failed status[1] when we look at the heat resource output we can see a rebalance was tried but it failed because it runs during the minimum number of hours before a partition can be reassigned[2]  see more  details in attached file [see attached file:  stack_update_heat_resource_output.txt]

Expect to see:
- update stack works without issues
- if rebalance cannot happen because it is not necessary or if the update runs during the minimum number of hours, the update process should be able to determine that and do not fail.


[1]
 [stack@716091-lab1-director ~]$ openstack overcloud update stack overcloud -i --templates ~/templates/my-overcloud -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e ~/templates/network-environment.yaml -e ~/templates/my-overcloud/environments/storage-environment.yaml 
starting package update on stack overcloud
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
FAILED
update finished with status FAILED


[2]
Swift::Ringbuilder::Rebalance[account]/Exec[rebalance_account]: Failed to call refresh: swift-ring-builder /etc/swift/account.builder rebalance 999 returned 1 instead of one of [0]
Swift::Ringbuilder::Rebalance[account]/Exec[rebalance_account]: swift-ring-builder /etc/swift/account.builder rebalance 999 returned 1 instead of one of [0]
Swift::Ringbuilder::Rebalance[container]/Exec[rebalance_container]: Failed to call refresh: swift-ring-builder /etc/swift/container.builder rebalance 999 returned 1 instead of one of [0]
Swift::Ringbuilder::Rebalance[container]/Exec[rebalance_container]: swift-ring-builder /etc/swift/container.builder rebalance 999 returned 1 instead of one of [0]
Swift::Ringbuilder::Rebalance[object]/Exec[rebalance_object]: Failed to call refresh: swift-ring-builder /etc/swift/object.builder rebalance 999 returned 1 instead of one of [0]
Swift::Ringbuilder::Rebalance[object]/Exec[rebalance_object]: swift-ring-builder /etc/swift/object.builder rebalance 999 returned 1 instead of one of [0]



Version-Release number of selected component (if applicable):
openstack-swift-2.3.0-4.el7ost.noarch 

Steps to reproduce: 
1. Deploy overcloud using OSP 7.0.3 from a director running OSP-D 7.2
2. Update director to 7.3 
3. Run openstack overcloud update stack with respective templates in interactive mode

NOTE: The important part here is that we deployed the 7.2 cloud, and 20-30 mins later we were running the update, but if you have a 7.0.3 overcloud running for more than that, try to rebalance before trying to update to 7.0.4 you can see the same rebalance results and the update might fail. [see attached file swift_rebalance_manual_test.txt]


Actual results:
fail

Expected results:
success install

Additional info:

1. We initially had undercloud OSP-D 7.2 and deployed an overcloud with OSP 7.0.3 or in other words, 7.2 images,  that worked without issues
2. Then we updated the undercloud to (OSP-D 7.3) and registered the overcloud nodes to our Satellite network.
3. We updated the undercloud to OSP-D 7.3  per the documentation,  no issues
4 We finally ran the stack update to update the current overcloud (this way the overcloud running OSP 7.0.3 would be updated to OSP 7.0.4) , that's then the error happens.

It's important to mention that we ran other tests and the updates were successful, it is just when there is nothing to rebalance that we got a failure.
Comment 3 Pete Zaitcev 2016-03-10 15:03:43 EST
If OSPd insists on running rebalances before the minimum number of
hours has expired, it needs to supply the "-f" flag.
Comment 7 Dave Maley 2016-05-03 12:49:30 EDT
customer is on osp7 and thus the fix provided in bug 1321088 will resolve this.

*** This bug has been marked as a duplicate of bug 1321088 ***

Note You need to log in before you can comment on or make changes to this bug.