Bug 645597 - getting multiple atom feed events when only one was expected
Summary: getting multiple atom feed events when only one was expected
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Candlepin
Classification: Community
Component: candlepin
Version: 0.5
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: ---
: ---
Assignee: Bryan Kearney
QA Contact: John Sefler
URL:
Whiteboard:
Depends On:
Blocks: Entitlement-Beta
TreeView+ depends on / blocked
 
Reported: 2010-10-21 22:22 UTC by John Sefler
Modified: 2015-05-14 15:23 UTC (History)
2 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2011-02-23 21:51:33 UTC
Embargoed:

Attachments (Terms of Use)
atom feed for the consumer 'admin' (10.87 KB, application/xml)
2010-11-11 16:34 UTC, Ajay Kumar Nadathur Sreenivasan 		no flags Details
Comment (362.88 KB, text/plain)
2011-01-06 19:57 UTC, John Sefler 		no flags Details
View All

Description John Sefler 2010-10-21 22:22:22 UTC 
Description of problem:
I have a nightly executed automated test PoolModifiedAndEntitlementModified_Test that runs against an on-premises server that has been failing for quite some time.  The failure occurs because a second "POOL MODIFIED" event is occurring when only one "POOL MODIFIED" event is expected. Here are the re-create steps:


1. Subscribe to available ProductName='RHEL for Physical Servers' ProductId='MKT-rhel-server'
subscription-manager-cli subscribe --pool=ff8080812bd07e4f012bd07fbc2100cb

2. Go into the candlepin database and update the pool by subtracting a month from its start date
Executing SQL: update cp_subscription set startdate='2010-09-20' where id=(select pool.subscriptionid from cp_pool pool where pool.id='ff8080812bd07e4f012bd07fbc2100cb'); 
Note: As expected, this SQL returned 1 meaning only one table row was updated.

3. refresh the pools
curl command: /usr/bin/curl -k -u admin:admin --request PUT https://jsefler-f12-candlepin.usersys.redhat.com:8443/candlepin/owners/admin/subscriptions

4. Analyze the new events on the atom feed for owner admin
https://jsefler-f12-candlepin.usersys.redhat.com:8443/candlepin/owners/admin/atom

The newest events that get added after refreshing the pools are:
ENTITLEMENT MODIFIED   <-- Expected since consumer is consuming MKT-rhel-server
POOL MODIFIED <-- Expected since we modified the pool startdate
POOL MODIFIED <-- NOT Expected - THIS IS THE BUG.  WHY IS THIS POOL MODIFIED 

Analyzing the raw atom feed below, we find the newest event, ENTITLEMENT MODIFIED, followed by its prior event POOL MODIFIED entityId=ff8080812bd07e4f012bd07fbc2100cb which is the expected "RHEL for Physical Servers" and the third event POOL MODIFIED entityId=ff8080812bd07e4f012bd07fbd8400d6 which is the "IPA Client Pack (50)" MKT-ipa-client-50 pool.  This third event is unexpected.  The "IPA Client Pack (50)" pool was NOT modified, NOT subscribed to, NOT supposed to be involved at all.  So why was an event tripped for this pool?  That's the bug.


Here is the raw atom feed:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><atom:feed xmlns:atom="http://www.w3.org/2005/Atom"><atom:title>Event feed for owner admin</atom:title><atom:updated>2010-10-21T16:38:49.812-04:00</atom:updated><atom:entry><atom:title>ENTITLEMENT MODIFIED</atom:title><atom:published>2010-10-21T16:38:49.812-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd083b02a0115</consumerId><entityId>ff8080812bd07e4f012bd083dcd20119</entityId><id>ff8080812bd07e4f012bd083f306011e</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>ENTITLEMENT</target><timestamp>2010-10-21T16:38:49.812-04:00</timestamp><type>MODIFIED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL MODIFIED</atom:title><atom:published>2010-10-21T16:38:49.619-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbc2100cb</entityId><id>ff8080812bd07e4f012bd083f310011f</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:38:49.619-04:00</timestamp><type>MODIFIED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL MODIFIED</atom:title><atom:published>2010-10-21T16:38:49.588-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbd8400d6</entityId><id>ff8080812bd07e4f012bd083f209011c</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:38:49.588-04:00</timestamp><type>MODIFIED</type></event></atom:content></atom:entry><atom:entry><atom:title>ENTITLEMENT CREATED</atom:title><atom:published>2010-10-21T16:38:44.383-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd083b02a0115</consumerId><entityId>ff8080812bd07e4f012bd083dcd20119</entityId><id>ff8080812bd07e4f012bd083ddad011b</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>ConsumerPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, uuid=3dc85667-943c-4187-9eb2-72358bc1be02, roles=[CONSUMER]]</principal><target>ENTITLEMENT</target><timestamp>2010-10-21T16:38:44.383-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>CONSUMER CREATED</atom:title><atom:published>2010-10-21T16:38:36.428-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd083b02a0115</consumerId><entityId>ff8080812bd07e4f012bd083b02a0115</entityId><id>ff8080812bd07e4f012bd083be9b0118</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=testuser1, roles=[OWNER_ADMIN]]</principal><target>CONSUMER</target><timestamp>2010-10-21T16:38:36.428-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>CONSUMER DELETED</atom:title><atom:published>2010-10-21T16:38:22.359-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd082114d0110</consumerId><entityId>ff8080812bd07e4f012bd082114d0110</entityId><id>ff8080812bd07e4f012bd08387b40114</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>ConsumerPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, uuid=41d6ebae-2908-48b0-a5d4-5276f8c2af7f, roles=[CONSUMER]]</principal><target>CONSUMER</target><timestamp>2010-10-21T16:38:22.359-04:00</timestamp><type>DELETED</type></event></atom:content></atom:entry><atom:entry><atom:title>CONSUMER CREATED</atom:title><atom:published>2010-10-21T16:36:50.168-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd082114d0110</consumerId><entityId>ff8080812bd07e4f012bd082114d0110</entityId><id>ff8080812bd07e4f012bd0821f890113</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=testuser1, roles=[OWNER_ADMIN]]</principal><target>CONSUMER</target><timestamp>2010-10-21T16:36:50.168-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>CONSUMER DELETED</atom:title><atom:published>2010-10-21T16:35:39.598-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd080c85300ed</consumerId><entityId>ff8080812bd07e4f012bd080c85300ed</entityId><id>ff8080812bd07e4f012bd0810bf1010f</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>ConsumerPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, uuid=5a8dd346-04c2-44a5-b293-e73cd375ae3b, roles=[CONSUMER]]</principal><target>CONSUMER</target><timestamp>2010-10-21T16:35:39.598-04:00</timestamp><type>DELETED</type></event></atom:content></atom:entry><atom:entry><atom:title>ENTITLEMENT DELETED</atom:title><atom:published>2010-10-21T16:35:39.446-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd080c85300ed</consumerId><entityId>ff8080812bd07e4f012bd080e1de00f1</entityId><id>ff8080812bd07e4f012bd0810b67010e</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>ConsumerPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, uuid=5a8dd346-04c2-44a5-b293-e73cd375ae3b, roles=[CONSUMER]]</principal><target>ENTITLEMENT</target><timestamp>2010-10-21T16:35:39.446-04:00</timestamp><type>DELETED</type></event></atom:content></atom:entry><atom:entry><atom:title>SUBSCRIPTION CREATED</atom:title><atom:published>2010-10-21T16:35:36.007-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd080fd370107</entityId><id>ff8080812bd07e4f012bd080fdeb0109</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>SUBSCRIPTION</target><timestamp>2010-10-21T16:35:36.007-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>EXPORT CREATED</atom:title><atom:published>2010-10-21T16:35:31.984-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd080c85300ed</consumerId><entityId>ff8080812bd07e4f012bd080c85300ed</entityId><id>ff8080812bd07e4f012bd080ee2e00f4</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>EXPORT</target><timestamp>2010-10-21T16:35:31.984-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>ENTITLEMENT CREATED</atom:title><atom:published>2010-10-21T16:35:29.187-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd080c85300ed</consumerId><entityId>ff8080812bd07e4f012bd080e1de00f1</entityId><id>ff8080812bd07e4f012bd080e33b00f3</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>ConsumerPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, uuid=5a8dd346-04c2-44a5-b293-e73cd375ae3b, roles=[CONSUMER]]</principal><target>ENTITLEMENT</target><timestamp>2010-10-21T16:35:29.187-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>CONSUMER CREATED</atom:title><atom:published>2010-10-21T16:35:24.475-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd080c85300ed</consumerId><entityId>ff8080812bd07e4f012bd080c85300ed</entityId><id>ff8080812bd07e4f012bd080d0d300f0</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=testuser1, roles=[OWNER_ADMIN]]</principal><target>CONSUMER</target><timestamp>2010-10-21T16:35:24.475-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>CONSUMER DELETED</atom:title><atom:published>2010-10-21T16:35:21.104-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd08068b100e3</consumerId><entityId>ff8080812bd07e4f012bd08068b100e3</entityId><id>ff8080812bd07e4f012bd080c3b900ec</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>ConsumerPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, uuid=71153a76-2415-4d17-9fce-a58dd653a0df, roles=[CONSUMER]]</principal><target>CONSUMER</target><timestamp>2010-10-21T16:35:21.104-04:00</timestamp><type>DELETED</type></event></atom:content></atom:entry><atom:entry><atom:title>CONSUMER CREATED</atom:title><atom:published>2010-10-21T16:35:03.318-04:00</atom:published><atom:content><event><consumerId>ff8080812bd07e4f012bd08068b100e3</consumerId><entityId>ff8080812bd07e4f012bd08068b100e3</entityId><id>ff8080812bd07e4f012bd0807e4600e6</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=testuser1, roles=[OWNER_ADMIN]]</principal><target>CONSUMER</target><timestamp>2010-10-21T16:35:03.318-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:14.027-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbd8400d6</entityId><id>ff8080812bd07e4f012bd07fbec400e2</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:14.027-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:14.014-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbd7800d5</entityId><id>ff8080812bd07e4f012bd07fbebf00e1</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:14.014-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:14-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbd6700d4</entityId><id>ff8080812bd07e4f012bd07fbeba00e0</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:14-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.980-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbd4900d3</entityId><id>ff8080812bd07e4f012bd07fbeb200df</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.980-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.937-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbd2300d2</entityId><id>ff8080812bd07e4f012bd07fbea800de</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.937-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.912-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbd1300d1</entityId><id>ff8080812bd07e4f012bd07fbea400dd</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.912-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.902-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbd0600d0</entityId><id>ff8080812bd07e4f012bd07fbe9e00dc</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.902-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.884-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbcf600cf</entityId><id>ff8080812bd07e4f012bd07fbe9800db</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.884-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.868-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbce700ce</entityId><id>ff8080812bd07e4f012bd07fbe9000da</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.868-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.857-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbcdd00cd</entityId><id>ff8080812bd07e4f012bd07fbe7f00d9</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.857-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.840-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbcbe00cc</entityId><id>ff8080812bd07e4f012bd07fbe7800d8</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.840-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry><atom:entry><atom:title>POOL CREATED</atom:title><atom:published>2010-10-21T16:34:13.784-04:00</atom:published><atom:content><event><entityId>ff8080812bd07e4f012bd07fbc2100cb</entityId><id>ff8080812bd07e4f012bd07fbe7200d7</id><ownerId>ff8080812bd07e4f012bd07e76080005</ownerId><principal>UserPrincipal [ownerId=ff8080812bd07e4f012bd07e76080005, username=admin, roles=[SUPER_ADMIN]]</principal><target>POOL</target><timestamp>2010-10-21T16:34:13.784-04:00</timestamp><type>CREATED</type></event></atom:content></atom:entry></atom:feed>



Version-Release number of selected component (if applicable):
This has been happening for the past few months:
[root@jsefler-f12-candlepin candlepin]# git show-ref master
5714d8726dbff903c0928c21ff55abc3be35f03a refs/heads/master
5714d8726dbff903c0928c21ff55abc3be35f03a refs/remotes/origin/master


Additional Info:
I spent time today walking through the failure with jesusr
We reproduced the failure, but could not find an explanation.
Comment 1 Ajay Kumar Nadathur Sreenivasan 2010-11-11 16:34:03 UTC 
Hi,
     I tried going through the same steps a couple of times using the latest as well as the ref provided in the bug report. 

========= steps below ================

sudo ./subscription-manager-cli register --username=admin --password=admin --forc
b70f460f-276b-4956-9aff-14c6e6f47165 admin
[ajay@garuda-rh src{master}]$ sudo ./subscription-manager-cli list --available | more+-------------------------------------------+
    Available Subscriptions
+-------------------------------------------+


ProductName:            RHEL for Physical Servers
ProductId:              MKT-rhel-server          
PoolId:                 ff8080812c3bbb85012c3bbc9bfb00e0
Quantity:               10                       
Expires:                2011-11-10               

.....

[ajay@garuda-rh src{master}]$ sudo ./subscription-manager-cli subscribe --pool ff8080812c3bbb85012c3bbc9bfb00e0

[ajay@garuda-rh proxy{pool_refresh}]$ curl -k -u admin:admin --request PUT https://localhost:8443/candlepin/owners/admin/subscriptions

[ajay@garuda-rh proxy{pool_refresh}]$ curl -k -u admin:admin --request GET "https://localhost:8443/candlepin/owners/admin/atom" > /tmp/atom.xml; xmllint --format /tmp/atom.xml > /tmp/atom1.xml

 ==============================================================
The atom file has been attached for your reference.
Comment 2 Ajay Kumar Nadathur Sreenivasan 2010-11-11 16:34:57 UTC 
Created attachment 459795 [details]
atom feed for the consumer 'admin'
Comment 3 Ajay Kumar Nadathur Sreenivasan 2010-11-11 16:36:09 UTC 
Forgot to mention the query used to update db.
I changed the dates using the following sql:

update cp_subscription 
set startdate='2011-07-20'
 where id = (
 		select pool.subscriptionid 
 		from cp_pool pool 
 		where pool.id='ff8080812c3bbb85012c3bbc9bfb00e0');
Comment 5 John Sefler 2011-01-06 19:57:52 UTC 
Created attachment 915180 [details]
Comment

(This comment was longer than 65,535 characters and has been moved to an attachment by Red Hat Bugzilla).
Note You need to log in before you can comment on or make changes to this bug.