Bug 710666

Summary: Delete of corrupted images fails
Product: [Retired] CloudForms Cloud Engine Reporter: Pete Zaitcev <zaitcev>
Component: iwhdAssignee: Pete Zaitcev <zaitcev>
Status: CLOSED CURRENTRELEASE QA Contact: Martin Kočí <mkoci>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 1.0.0CC: dajohnso, deltacloud-maint, jguiditt, mkoci, ssachdev
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-02-01 19:25:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
proposed patch
none
shell script testing corruption none

Description Pete Zaitcev 2011-06-03 23:52:44 UTC
Description of problem:

If image or bucket is missing in the filesystem back-end, an attempt
to delete an image fails. So, if anything bad happens to back-end,
the database cannot "heal".

Version-Release number of selected component (if applicable):

0.95

How reproducible:

Synchronous

Steps to Reproduce:
1. Create a bucket and an image
2. delete image and bucket using Linux shell (imitate corruption)
3. Attempt to delete the image
  
Actual results:

Deletion fails with "500 Internal Error"

Expected results:

The image record should be deleted from the database.

Additional info:

Perhaps what we really need is some kind of automatic consistency
checker. Not only disappearing images need to be recovered, but also
images with bit errors. This must account for replication, too.
For now though simple things ought to continue to work in case
of consistency issues.

Comment 1 wes hayutin 2011-06-16 18:35:35 UTC
please recreate and paste build info

Comment 2 Dave Johnson 2011-06-16 21:11:17 UTC
Doing a src install from iwhd git repo commit 9e86ebf939a68c37c04004e8248013aab8f8cbb9 shows that this is still a issue although ***sligthly different***.  The delete through the API returns 404 and a subsequent get receives a 500.

http://hudson.rhq.lab.eng.bos.redhat.com:8080/hudson/job/imageWarehouse/758/
http://hudson.rhq.lab.eng.bos.redhat.com:8080/hudson/job/imageWarehouse/758/testngreports

Steps to reproduce
1. create a bucket, curl -X POST -d 'name=test_bucket_2' http://ip:port/_new
2. create a object in the bucket, curl -X PUT http://ip:port/test_bucket_2/camaro
3. rm -rf /var/lib/iwhd/<bucket>/<object>, rm -rf /var/lib/iwhd/test_bucket_2/camaro
4. try to delete object through the api, curl -X DELETE http://ip:port/test_bucket_2/camaro  (receive 404 error)
5. http get on the object, curl -X GET http://ip:port/test_bucket_2/camaro (receive 500 error)

Comment 3 Pete Zaitcev 2011-06-16 23:19:37 UTC
Gentlemen, I just wanted to have a bug so that I don't forget to fix it.
It is trivially reproducible in iwhd-0.96.

Comment 4 Pete Zaitcev 2011-07-24 23:16:41 UTC
Created attachment 514942 [details]
proposed patch

Comment 5 Dave Johnson 2011-08-10 15:33:17 UTC
I think there is still an issue here, issuing get on a key that was corrupted and then deleted through curl returns a 500, Internal Server Error.

==================================
Test delete on corrupted key
==================================
create bucket, rc: 201
create a key in the bucket, rc: 201
remove the key on fs, rc: 0
delete key w/ curl, rc: 404
get key w/ curl, expect rc404, rc: 500
cleanup bucket, rc: 404

Comment 6 Dave Johnson 2011-08-10 15:34:49 UTC
Created attachment 517644 [details]
shell script testing corruption

Comment 7 Pete Zaitcev 2011-08-10 23:11:50 UTC
Oh it is there all right. Missed 0.98.

Comment 9 wes hayutin 2011-09-28 16:39:53 UTC
making sure all the bugs are at the right version for future queries

Comment 11 Martin Kočí 2011-11-07 10:25:43 UTC
Test passed with version 0.991. I will test it against 1.0 yet and create automatic test for regression testing. 

==================================
Test delete on corrupted bucket
==================================
create bucket, rc: 201create a object in the bucket
remove the object of th bucket on fs, rc: 0
delete the bucket w/ curl, rc: 200
get bucket w/ curl, expect rc404, rc: 404

==================================
Test delete on corrupted key
==================================
create bucket, rc: 201
create a key in the bucket, rc: 201
remove the key on fs, rc: 0
delete key w/ curl, rc: 200
get key w/ curl, expect rc404, rc: 404
cleanup bucket, rc: 404

Comment 12 Martin Kočí 2011-11-07 11:55:52 UTC
Successfully tested with 1.0 version too. Moving the bug to VERIFIED.

# ./test.sh && iwhd --version
==================================
Test delete on corrupted bucket
==================================
create bucket, rc: 201
create an object in the bucket, rc: 201
remove the object of th bucket on fs, rc: 0
delete the bucket w/ curl, rc: 200
get bucket w/ curl, expect rc 404, rc: 404

==================================
Test delete on corrupted key
==================================
create bucket, rc: 201
create a key in the bucket, rc: 201
remove the key on fs, rc: 0
delete key w/ curl, rc: 200
get key w/ curl, expect rc 404, rc: 404
cleanup bucket, rc: 404

iwhd (iwhd) 1.0.1-1945