Bug 645597

Summary: getting multiple atom feed events when only one was expected
Product: [Community] Candlepin Reporter: John Sefler <jsefler>
Component: candlepinAssignee: Bryan Kearney <bkearney>
Status: CLOSED CURRENTRELEASE QA Contact: John Sefler <jsefler>
Severity: medium Docs Contact:
Priority: low    
Version: 0.5CC: bkearney, whayutin
Target Milestone: ---Keywords: QA-Closed
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-02-23 21:51:33 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 639436    
Attachments:
Description Flags
atom feed for the consumer 'admin'
none
Comment none

Description John Sefler 2010-10-21 22:22:22 UTC 
Description of problem:
I have a nightly executed automated test PoolModifiedAndEntitlementModified_Test that runs against an on-premises server that has been failing for quite some time.  The failure occurs because a second "POOL MODIFIED" event is occurring when only one "POOL MODIFIED" event is expected. Here are the re-create steps:


1. Subscribe to available ProductName='RHEL for Physical Servers' ProductId='MKT-rhel-server'
subscription-manager-cli subscribe --pool=ff8080812bd07e4f012bd07fbc2100cb

2. Go into the candlepin database and update the pool by subtracting a month from its start date
Executing SQL: update cp_subscription set startdate='2010-09-20' where id=(select pool.subscriptionid from cp_pool pool where pool.id='ff8080812bd07e4f012bd07fbc2100cb'); 
Note: As expected, this SQL returned 1 meaning only one table row was updated.

3. refresh the pools
curl command: /usr/bin/curl -k -u admin:admin --request PUT https://jsefler-f12-candlepin.usersys.redhat.com:8443/candlepin/owners/admin/subscriptions

4. Analyze the new events on the atom feed for owner admin
https://jsefler-f12-candlepin.usersys.redhat.com:8443/candlepin/owners/admin/atom

The newest events that get added after refreshing the pools are:
ENTITLEMENT MODIFIED   <-- Expected since consumer is consuming MKT-rhel-server
POOL MODIFIED <-- Expected since we modified the pool startdate
POOL MODIFIED <-- NOT Expected - THIS IS THE BUG.  WHY IS THIS POOL MODIFIED 

Analyzing the raw atom feed below, we find the newest event, ENTITLEMENT MODIFIED, followed by its prior event POOL MODIFIED entityId=ff8080812bd07e4f012bd07fbc2100cb which is the expected "RHEL for Physical Servers" and the third event POOL MODIFIED entityId=ff8080812bd07e4f012bd07fbd8400d6 which is the "IPA Client Pack (50)" MKT-ipa-client-50 pool.  This third event is unexpected.  The "IPA Client Pack (50)" pool was NOT modified, NOT subscribed to, NOT supposed to be involved at all.  So why was an event tripped for this pool?  That's the bug.


Here is the raw atom feed:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><atom:feed xmlns:atom="http://www.w3.org/2005/Atom"><atom:title>Event feed for owner admin</atom:title><atom:updated>2010-10-21T16:38:49.812-04:00</atom:updated><atom:entry><atom:title>ENTITLEMENT MODIFIED</atom:title><atom:published>2010-10-21T16:38:49.812-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd083b02a0115</consumerId><entityId>ff8080812bd07e4f012bd083dcd20119</entityId><id>ff8080812bd07e4f012bd083f306011e</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>ENTITLEMENT</target><timestamp>2010-10-21T16:38:49.812-04:00</timestamp><type>MODIFIED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL MODIFIED</atom:title><atom:published>2010-10-21T16:38:49.619-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbc2100cb</entityId><id>ff8080812bd07e4f012bd083f310011f</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:38:49.619-04:00</timestamp><type>MODIFIED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL MODIFIED</atom:title><atom:published>2010-10-21T16:38:49.588-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbd8400d6</entityId><id>ff8080812bd07e4f012bd083f209011c</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:38:49.588-04:00</timestamp><type>MODIFIED</type></event></atom:content></atom:entry><atom:entry><atom:title>ENTITLEMENT CREATED</atom:title><atom:published>2010-10-21T16:38:44.383-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd083b02a0115</consumerId><entityId>ff8080812bd07e4f012bd083dcd20119</entityId><id>ff8080812bd07e4f012bd083ddad011b</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>ConsumerPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, uuid=3dc85667-943c-4187-9eb2-72358bc1be02, roles=[CONSUMER]]</principal><target>ENTITLEMENT</target><timestamp>2010-10-21T16:38:44.383-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>CONSUMER CREATED</atom:title><atom:published>2010-10-21T16:38:36.428-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd083b02a0115</consumerId><entityId>ff8080812bd07e4f012bd083b02a0115</entityId><id>ff8080812bd07e4f012bd083be9b0118</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=testuser1, roles=[OWNER_ADMIN]]</principal><target>CONSUMER</target><timestamp>2010-10-21T16:38:36.428-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>CONSUMER DELETED</atom:title><atom:published>2010-10-21T16:38:22.359-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd082114d0110</consumerId><entityId>ff8080812bd07e4f012bd082114d0110</entityId><id>ff8080812bd07e4f012bd08387b40114</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>ConsumerPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, uuid=41d6ebae-2908-48b0-a5d4-5276f8c2af7f, roles=[CONSUMER]]</principal><target>CONSUMER</target><timestamp>2010-10-21T16:38:22.359-04:00</timestamp><type>DELETED</type></event></atom:content></atom:entry><atom:entry><atom:title>CONSUMER CREATED</atom:title><atom:published>2010-10-21T16:36:50.168-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd082114d0110</consumerId><entityId>ff8080812bd07e4f012bd082114d0110</entityId><id>ff8080812bd07e4f012bd0821f890113</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=testuser1, roles=[OWNER_ADMIN]]</principal><target>CONSUMER</target><timestamp>2010-10-21T16:36:50.168-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>CONSUMER DELETED</atom:title><atom:published>2010-10-21T16:35:39.598-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd080c85300ed</consumerId><entityId>ff8080812bd07e4f012bd080c85300ed</entityId><id>ff8080812bd07e4f012bd0810bf1010f</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>ConsumerPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, uuid=5a8dd346-04c2-44a5-b293-e73cd375ae3b, roles=[CONSUMER]]</principal><target>CONSUMER</target><timestamp>2010-10-21T16:35:39.598-04:00</timestamp><type>DELETED</type></event></atom:content></atom:entry><atom:entry><atom:title>ENTITLEMENT DELETED</atom:title><atom:published>2010-10-21T16:35:39.446-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd080c85300ed</consumerId><entityId>ff8080812bd07e4f012bd080e1de00f1</entityId><id>ff8080812bd07e4f012bd0810b67010e</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>ConsumerPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, uuid=5a8dd346-04c2-44a5-b293-e73cd375ae3b, roles=[CONSUMER]]</principal><target>ENTITLEMENT</target><timestamp>2010-10-21T16:35:39.446-04:00</timestamp><type>DELETED</type></event></atom:content></atom:entry><atom:entry><atom:title>SUBSCRIPTION CREATED</atom:title><atom:published>2010-10-21T16:35:36.007-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd080fd370107</entityId><id>ff8080812bd07e4f012bd080fdeb0109</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>SUBSCRIPTION</target><timestamp>2010-10-21T16:35:36.007-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>EXPORT CREATED</atom:title><atom:published>2010-10-21T16:35:31.984-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd080c85300ed</consumerId><entityId>ff8080812bd07e4f012bd080c85300ed</entityId><id>ff8080812bd07e4f012bd080ee2e00f4</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>EXPORT</target><timestamp>2010-10-21T16:35:31.984-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>ENTITLEMENT CREATED</atom:title><atom:published>2010-10-21T16:35:29.187-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd080c85300ed</consumerId><entityId>ff8080812bd07e4f012bd080e1de00f1</entityId><id>ff8080812bd07e4f012bd080e33b00f3</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>ConsumerPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, uuid=5a8dd346-04c2-44a5-b293-e73cd375ae3b, roles=[CONSUMER]]</principal><target>ENTITLEMENT</target><timestamp>2010-10-21T16:35:29.187-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>CONSUMER CREATED</atom:title><atom:published>2010-10-21T16:35:24.475-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd080c85300ed</consumerId><entityId>ff8080812bd07e4f012bd080c85300ed</entityId><id>ff8080812bd07e4f012bd080d0d300f0</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=testuser1, roles=[OWNER_ADMIN]]</principal><target>CONSUMER</target><timestamp>2010-10-21T16:35:24.475-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>CONSUMER DELETED</atom:title><atom:published>2010-10-21T16:35:21.104-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd08068b100e3</consumerId><entityId>ff8080812bd07e4f012bd08068b100e3</entityId><id>ff8080812bd07e4f012bd080c3b900ec</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>ConsumerPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, uuid=71153a76-2415-4d17-9fce-a58dd653a0df, roles=[CONSUMER]]</principal><target>CONSUMER</target><timestamp>2010-10-21T16:35:21.104-04:00</timestamp><type>DELETED</type></event></atom:content></atom:entry><atom:entry><atom:title>CONSUMER CREATED</atom:title><atom:published>2010-10-21T16:35:03.318-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd08068b100e3</consumerId><entityId>ff8080812bd07e4f012bd08068b100e3</entityId><id>ff8080812bd07e4f012bd0807e4600e6</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=testuser1, roles=[OWNER_ADMIN]]</principal><target>CONSUMER</target><timestamp>2010-10-21T16:35:03.318-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:14.027-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbd8400d6</entityId><id>ff8080812bd07e4f012bd07fbec400e2</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:14.027-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:14.014-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbd7800d5</entityId><id>ff8080812bd07e4f012bd07fbebf00e1</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:14.014-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:14-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbd6700d4</entityId><id>ff8080812bd07e4f012bd07fbeba00e0</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:14-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.980-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbd4900d3</entityId><id>ff8080812bd07e4f012bd07fbeb200df</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.980-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.937-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbd2300d2</entityId><id>ff8080812bd07e4f012bd07fbea800de</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.937-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.912-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbd1300d1</entityId><id>ff8080812bd07e4f012bd07fbea400dd</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.912-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.902-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbd0600d0</entityId><id>ff8080812bd07e4f012bd07fbe9e00dc</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.902-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.884-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbcf600cf</entityId><id>ff8080812bd07e4f012bd07fbe9800db</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.884-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.868-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbce700ce</entityId><id>ff8080812bd07e4f012bd07fbe9000da</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.868-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.857-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbcdd00cd</entityId><id>ff8080812bd07e4f012bd07fbe7f00d9</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.857-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.840-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbcbe00cc</entityId><id>ff8080812bd07e4f012bd07fbe7800d8</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.840-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.784-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbc2100cb</entityId><id>ff8080812bd07e4f012bd07fbe7200d7</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.784-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry></atom:feed>



Version-Release number of selected component (if applicable):
This has been happening for the past few months:
[root@jsefler-f12-candlepin candlepin]# git show-ref master
5714d8726dbff903c0928c21ff55abc3be35f03a refs/heads/master
5714d8726dbff903c0928c21ff55abc3be35f03a refs/remotes/origin/master


Additional Info:
I spent time today walking through the failure with jesusr
We reproduced the failure, but could not find an explanation.
Comment 1 Ajay Kumar Nadathur Sreenivasan 2010-11-11 16:34:03 UTC 
Hi,
     I tried going through the same steps a couple of times using the latest as well as the ref provided in the bug report. 

========= steps below ================

sudo ./subscription-manager-cli register --username=admin --password=admin --forc
b70f460f-276b-4956-9aff-14c6e6f47165 admin
[ajay@garuda-rh src{master}]$ sudo ./subscription-manager-cli list --available | more+-------------------------------------------+
    Available Subscriptions
+-------------------------------------------+


ProductName:            RHEL for Physical Servers
ProductId:              MKT-rhel-server          
PoolId:                 ff8080812c3bbb85012c3bbc9bfb00e0
Quantity:               10                       
Expires:                2011-11-10               

.....

[ajay@garuda-rh src{master}]$ sudo ./subscription-manager-cli subscribe --pool ff8080812c3bbb85012c3bbc9bfb00e0

[ajay@garuda-rh proxy{pool_refresh}]$ curl -k -u admin:admin --request PUT https://localhost:8443/candlepin/owners/admin/subscriptions

[ajay@garuda-rh proxy{pool_refresh}]$ curl -k -u admin:admin --request GET "https://localhost:8443/candlepin/owners/admin/atom" > /tmp/atom.xml; xmllint --format /tmp/atom.xml > /tmp/atom1.xml

 ==============================================================
The atom file has been attached for your reference.
Comment 2 Ajay Kumar Nadathur Sreenivasan 2010-11-11 16:34:57 UTC 
Created attachment 459795 [details]
atom feed for the consumer 'admin'
Comment 3 Ajay Kumar Nadathur Sreenivasan 2010-11-11 16:36:09 UTC 
Forgot to mention the query used to update db.
I changed the dates using the following sql:

update cp_subscription 
set startdate='2011-07-20'
 where id = (
 		select pool.subscriptionid 
 		from cp_pool pool 
 		where pool.id='ff8080812c3bbb85012c3bbc9bfb00e0');
Comment 5 John Sefler 2011-01-06 19:57:52 UTC 
Created attachment 915180 [details]
Comment

(This comment was longer than 65,535 characters and has been moved to an attachment by Red Hat Bugzilla).