Hide Forgot
Description of problem: Once in a while, an image just disappears. Everything kept in Mongo remains, so all attributes are accessible, but the body is just gone. Version-Release number of selected component (if applicable): iwhd-1.2 How reproducible: Very difficult Steps to Reproduce: No known steps Actual results: Image body not available Expected results: Consistency between attributes and image Additional info: Richard Su hit it before, but in his testing there's never a certainty, because of buggy Puppet scripts that relocate the back-end repository. Scott Seago seems to have found a clear case.
Created attachment 548674 [details] Scott's saved iwhd.log
Created attachment 548677 [details] Scott's current iwhd.log The offending image name is 98575b00-2025-11e1-8f40-52540042446c.
Unfortunately this is also unclean, because there was some tampering with the backing store. It goes like so. The last we heard from our image is this: Dec 08 12:12:02 iwhd[31534]: 127.0.0.1.44386 "GET /images/98575b00-2025-11e1-8f40-52540042446c" 0 Dec 08 12:12:02 iwhd[31534]: 127.0.0.1.44386 200 33 So far so good, but then this happens: Dec 09 02:32:45 iwhd[31534]: 127.0.0.1.49914 "PUT /templates" - /usr/bin/iwhd: templates: failed to create directory: File exists Dec 09 02:32:45 iwhd[31534]: 127.0.0.1.49914 500 0 Initializing... 0 replication servers defined Dec 09 02:36:34 iwhd[18778]: convert-provider: ITER key: name Dec 09 02:36:34 iwhd[18778]: convert-provider: ITER key: type Dec 09 02:36:34 iwhd[18778]: convert-provider: ITER key: path Dec 09 02:36:34 iwhd[18778]: 127.0.0.1.49972 "GET /" - Dec 09 02:36:34 iwhd[18778]: 127.0.0.1.49972 200 260 Dec 09 02:36:34 iwhd[18778]: 127.0.0.1.49973 "PUT /templates" - Dec 09 02:36:34 iwhd[18778]: 127.0.0.1.49973 201 0 So, bucket exists, then server restarts and suddenly the bucket can be created. This only can be if someone went and removed it from the backing store with rmdir(1). Then a few day pass without any activity (over the weekend actually), and then immediately: Dec 13 12:55:08 iwhd[22053]: 127.0.0.1.54491 "GET /images/98575b00-2025-11e1-8f40-52540042446c" 0 Dec 13 12:55:08 iwhd[22053]: 127.0.0.1.54491 500 0 The first thing we see is object gone from the backing store. I suspect Scott accidentially deleted it after all.
Hi Pete, It looks like we can put this to rest, now. Ok to close?