Red Hat Bugzilla – Bug 1013594
swift: conflicting commands between horizon and cli cause issues with containers
Last modified: 2016-04-26 12:27:36 EDT
Created attachment 805170 [details]
Description of problem:
I uploaded a file in cli while deleting the container from horizon.
it seems that the container is there but we cannot retrieve the files and when we try to delete the container from swift cli we get an error.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. delete swift empty container from horizon while uploading file to it in cli
it appears that the container is not deleted but we cannot list objects or delete it
we should have a locking mechanized for conflicting commands
this is the upload command from cli while I was deleting the container in horizon:
[root@nott-vdsa tmp(keystone_admin)]# swift upload bla packstack-answers-20130924-140637.txt
if we run swift list we can see the container but cannot delete it:
[root@nott-vdsa tmp(keystone_admin)]# swift list
[root@nott-vdsa tmp(keystone_admin)]# swift delete bla
Container 'bla' not found
list the container does not work as well:
[root@nott-vdsa tmp(keystone_admin)]# swift list bla
Container 'bla' not found
The only way to remove it is to run swift delete all
Yeah, I know... The usual workaround is to do swift post bla first,
then delete. You do not always want to blow every container with -a.
Created attachment 821367 [details]
a test script
I was unable to reproduce the problem thus far. I hit something similar
one time, but it was related to poor cluster health.
Dafna: I understand you're busy, but do you happen to have a reproducer
for this? I need to know what the health of the cluster was when it happened,
in particular if there was an out of space condition. My tests show that
we primarily rely on updaters to resolve these inconsistencies (not auditors),
so running out of space in the "pending" directory would cause havoc.
I did have a case of low space on my data servers.
I don't have that setup any more and the setup I have has 1 zone (the other one had 2 zones and 5 data servers).
I think that the only way to reproduce this is if the delete will take a few more seconds than it does in the 1 zone setup I have now.
I did notice that we have a conflict error when we do things the other way around (delete from cli and upload from UI) but when I upload from cli and delete from horizon we will create the container if it does not exist.
is it possible that we have an exit in the code when deleting from cli but we do not have that in horizon?
[root@nott-vdsa ~(keystone_admin)]# swift delete bla
Container DELETE failed: http://10.35.64.69:8080/v1/AUTH_f316d4aeb27b43979c3ec3543ae37985/bla 409 Conflict [first 60 chars of response] <html><h1>Conflict</h1><p>There was a conflict when trying t
Closing with INSUFFICIENT_DATA.