Bug 2119145 - Stack update/upgrade fails with "Container DELETE failed" message raised from config-download workflow
Summary: Stack update/upgrade fails with "Container DELETE failed" message raised from...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 16.2 (Train)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z4
: 16.2 (Train on RHEL 8.4)
Assignee: Brendan Shephard
QA Contact: David Rosenfeld
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-08-17 16:56 UTC by Mark Jones
Modified: 2022-12-07 19:24 UTC (History)
9 users (show)

Fixed In Version: openstack-tripleo-common-11.7.1-2.20220923014727.c488b59.el8ost
Doc Type: Bug Fix
Doc Text:
Before this update, when the number of objects in the Object Storage service (swift) container for the overcloud exceeded 10,000, only the first 10,000 objects in the config-download directory were cleaned up during a delete operation, and the remaining objects prevented the container from getting deleted. With this update, there is added handling for cases where there are more than 10,000 objects in the Object Storage service container for the overcloud.
Clone Of:
Environment:
Last Closed: 2022-12-07 19:24:09 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 855118 0 None MERGED [Train-only] Ensure complete object list 2022-09-20 02:30:27 UTC
Red Hat Issue Tracker OSP-18245 0 None None None 2022-08-17 17:05:56 UTC
Red Hat Product Errata RHBA-2022:8794 0 None None None 2022-12-07 19:24:50 UTC

Comment 5 Brendan Shephard 2022-08-26 23:31:44 UTC
I submitted a patch for this so the number in the conf file should be irrelevant with this patch:
https://review.opendev.org/c/openstack/tripleo-common/+/854722

What are you doing to count them? Just something like find . | wc -l?  Or git count-objects?

$ git count-objects
1096 objects, 6612 kilobytes

That's from my tripleo environment with a Standlone node and a Compute node.
10k probably isn't unreasonable at that size.

I think that patch should be fine to address it and maybe we make a KCS for OSP13 since we won't be back porting that to Queens.

Comment 7 Brendan Shephard 2022-08-29 22:18:34 UTC
Hey,

I'm not sure if it's necessary to loop over the objects in the container deleting them to be honest. I think that might have just been a measure to work around a problem. 

I'll see if I can improve the efficiency of this process while we're going.

And yep, I'll back port it to 16.1 as well.

I'll just leave NEEDSINFO on me for now to keep it in my list.

Comment 24 David Rosenfeld 2022-11-14 13:31:51 UTC
Ran a script to create 10000 objects:

(undercloud) [stack@undercloud-0 ~]$ openstack object list overcloud -f value | wc -l
10000

Successfully performed stack update after script completed.

Comment 32 errata-xmlrpc 2022-12-07 19:24:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 16.2.4), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8794


Note You need to log in before you can comment on or make changes to this bug.