Bug 769062

Summary:

Image bodies disappear randomly

Product:

[Fedora] Fedora

Reporter:

Pete Zaitcev <zaitcev>

Component:

iwhd

Assignee:

Jim Meyering <meyering>

Status:

CLOSED NOTABUG

QA Contact:

Fedora Extras Quality Assurance <extras-qa>

Severity:

unspecified

Docs Contact:

Priority:

unspecified

Version:

rawhide

CC:

clalancette, meyering, zaitcev

Target Milestone:

---

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2012-04-03 21:21:43 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Attachments:

Description	Flags
Scott's saved iwhd.log	none
Scott's current iwhd.log	none

Description Pete Zaitcev 2011-12-19 19:41:42 UTC

Description of problem:

Once in a while, an image just disappears. Everything kept in Mongo
remains, so all attributes are accessible, but the body is just gone.

Version-Release number of selected component (if applicable):

iwhd-1.2

How reproducible:

Very difficult

Steps to Reproduce:
No known steps
  
Actual results:
Image body not available

Expected results:
Consistency between attributes and image

Additional info:
Richard Su hit it before, but in his testing there's never a certainty,
because of buggy Puppet scripts that relocate the back-end repository.
Scott Seago seems to have found a clear case.

Comment 1 Pete Zaitcev 2011-12-19 19:48:55 UTC

Created attachment 548674 [details]
Scott's saved iwhd.log

Comment 2 Pete Zaitcev 2011-12-19 19:49:45 UTC

Created attachment 548677 [details]
Scott's current iwhd.log

The offending image name is 98575b00-2025-11e1-8f40-52540042446c.

Comment 3 Pete Zaitcev 2012-01-02 18:22:16 UTC

Unfortunately this is also unclean, because there was some tampering
with the backing store.

It goes like so. The last we heard from our image is this:

Dec 08 12:12:02 iwhd[31534]: 127.0.0.1.44386 "GET /images/98575b00-2025-11e1-8f40-52540042446c" 0
Dec 08 12:12:02 iwhd[31534]: 127.0.0.1.44386 200 33

So far so good, but then this happens:

Dec 09 02:32:45 iwhd[31534]: 127.0.0.1.49914 "PUT /templates" -
/usr/bin/iwhd: templates: failed to create directory: File exists
Dec 09 02:32:45 iwhd[31534]: 127.0.0.1.49914 500 0
Initializing...
0 replication servers defined
Dec 09 02:36:34 iwhd[18778]: convert-provider: ITER key: name
Dec 09 02:36:34 iwhd[18778]: convert-provider: ITER key: type
Dec 09 02:36:34 iwhd[18778]: convert-provider: ITER key: path
Dec 09 02:36:34 iwhd[18778]: 127.0.0.1.49972 "GET /" -
Dec 09 02:36:34 iwhd[18778]: 127.0.0.1.49972 200 260
Dec 09 02:36:34 iwhd[18778]: 127.0.0.1.49973 "PUT /templates" -
Dec 09 02:36:34 iwhd[18778]: 127.0.0.1.49973 201 0

So, bucket exists, then server restarts and suddenly the bucket can
be created. This only can be if someone went and removed it from
the backing store with rmdir(1). Then a few day pass without any activity
(over the weekend actually), and then immediately:

Dec 13 12:55:08 iwhd[22053]: 127.0.0.1.54491 "GET /images/98575b00-2025-11e1-8f40-52540042446c" 0
Dec 13 12:55:08 iwhd[22053]: 127.0.0.1.54491 500 0

The first thing we see is object gone from the backing store.
I suspect Scott accidentially deleted it after all.

Comment 4 Jim Meyering 2012-04-03 08:08:01 UTC

Hi Pete,

It looks like we can put this to rest, now.  Ok to close?