Bug 855053 - [RFE] Change repo unit count to be per unit type
[RFE] Change repo unit count to be per unit type
Status: CLOSED CURRENTRELEASE
Product: Pulp
Classification: Community
Component: user-experience (Show other bugs)
Master
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 2.1.0
Assigned To: Michael Hrivnak
Preethi Thomas
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-06 11:03 EDT by Jay Dobies
Modified: 2013-09-09 12:30 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-04-08 11:59:01 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jay Dobies 2012-09-06 11:03:24 EDT
Currently this is a single counter in the repo. To the end user, this would be much more helpful if we could show it broken down by type.

I don't think this is impossible. All of the hooks to when the counter is changed are in the conduits. I'm 99% sure all of the calls into the conduits at those points will include the unit type.

We'd have to write a database upgrade script to fix existing repositories, but that's a handful of pretty simple mongo queries and some data collation.
Comment 1 Michael Hrivnak 2012-10-15 16:16:35 EDT
Thankfully, the hooks are very well defined. This is what would be required:

- RepoManager.update_unit_count() needs to take an additional parameter, content_type_id.
- That method needs to store per-type unit counts in a new subdocument where keys are content type ids. This is quite easy, and mongo's increment operator will auto-create the attribute and/or the subdocument if both don't exist.
- associate_unit_by_id, associate_all_by_ids, and unassociate_by_criteria (all on the RepoUnitAssociationManager) need to pass the content type ID to RepoManager.update_unit_count. This is easy, because each of those methods already knows the type.
- If we name the subdocument something besides "content_unit_count", there are a couple of commands where we would need to change this in the default filters.
- Update unit tests.
- optionally present a total content unit count, either by having the client calculate it on-the-fly, or by maintaining a "total" attribute in the subdocument. The latter would require two increment commands instead of one. 

Probably the most complicated part of this would be writing a migration to convert existing repos. The migration would have to load and inspect a tremendous portion of the data in the system. I suspect that would double the development work load of this task.

Since this task was to investigate adding the feature, I think it's done. Jay, can you clarify what should happen from here? Will you open a new issue to actually do the work?
Comment 2 Jay Dobies 2012-10-16 08:59:09 EDT
I don't actually think the upgrade script will be all that hard. I think that, per repo, you could do a query to retrieve a list of unique type IDs for content in the repo, then another query per type to do the count for each. Shouldn't be too bad if you just have pymongo do counts.

Given that you're pretty booked this sprint, I'm going to just move this bug to next sprint and morph it into the implementation. Sounds like it's more than an hour or two, so we'll make a story for it.
Comment 3 Michael Hrivnak 2013-03-01 18:32:31 EST
https://github.com/pulp/pulp/pull/359
Comment 4 Jeff Ortel 2013-03-06 10:20:42 EST
build: 2.1.0-0.20.beta
Comment 5 Jay Dobies 2013-03-06 16:49:28 EST
FYI when verifying, keep an eye out on upgraded installs. I'm not sure I can replicate the issue I saw and we did a second look at the code and it looks good, but keep it in mind:


<jdob> so if you have repos in 2.0.8 and then run pulp-manage-db for 2.1.0
<jdob> it'll split that apart: https://gist.github.com/jdob/5103244
<jdob> notice on line 10 there, it still has the old count
<jdob> that's supposed to be removed in the migration
<jdob> and it only shows up in repo search
<jdob> i'm thinking this is a one-off issue that i still have both
<jdob> but fyi, keep an eye out for it
<preethi> ok
<jdob> on second thought, I'll just dig up the RFE and include this note there


$ pulp-admin rpm repo search
+----------------------------------------------------------------------+
Repositories
+----------------------------------------------------------------------+
 
Id: f17
Display Name: f17
Description: None
Content Unit Count: 3584  # shouldn't be here
Content Unit Counts:
Distribution: 1
Package Category: 7
Package Group: 227
Rpm: 3349
Notes:
Comment 6 Preethi Thomas 2013-03-19 10:24:17 EDT
verified

[root@preethi-el6-pulp ~]# rpm -q pulp-server
pulp-server-2.1.0-0.23.beta.noarch
[root@preethi-el6-pulp ~]# 

Also tested on upgrade from 2.0.8 ->2.1

Id:                  zoo
Display Name:        zoo
Description:         None
Content Unit Counts: 
  Erratum:          4
  Package Category: 1
  Package Group:    2
  Rpm:              32
Notes:               
Importers:           
  Config:           
    Feed URL: http://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/
  Id:               yum_importer
  Importer Type Id: yum_importer
  Last Sync:        2013-01-31T08:49:04-05:00
  Repo Id:          zoo
  Scheduled Syncs:  
Distributors:        
  Auto Publish:        True
  Config:              
    Http:         False
    Https:        True
    Relative URL: /repos/pulp/pulp/demo_repos/zoo/
  Distributor Type Id: yum_distributor
  Id:                  yum_distributor
  Last Publish:        2013-01-31T08:49:06-05:00
  Repo Id:             zoo
  Scheduled Publishes: 
  Auto Publish:        False
  Config:              
    Http:  False
    Https: True
  Distributor Type Id: export_distributor
  Id:                  export_distributor
  Last Publish:        None
  Repo Id:             zoo
  Scheduled Publishes: 


[root@preethi-el6-pulp ~]# pulp-admin -u admin -p admin  repo list
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+

Id:                  rh6-epel-live
Display Name:        rh6-epel-live
Description:         None
Content Unit Counts: 
  Erratum:          2023
  Package Category: 3
  Package Group:    208
  Rpm:              8336

Id:                  uploads
Display Name:        uploads
Description:         None
Content Unit Counts: 
  Rpm: 4

Id:                  copy-epel
Display Name:        copy-epel
Description:         None
Content Unit Counts: 
  Rpm: 8336

Id:                  rhel6_3
Display Name:        rhel6_3
Description:         None
Content Unit Counts: 

Id:                  test
Display Name:        test
Description:         None
Content Unit Counts: 

Id:                  zoo
Display Name:        zoo
Description:         None
Content Unit Counts: 
  Erratum:          4
  Package Category: 1
  Package Group:    2
  Rpm:              32


[root@preethi-el6-pulp ~]# 


[root@preethi-el6-pulp ~]# pulp-admin rpm repo search
+----------------------------------------------------------------------+
                              Repositories
+----------------------------------------------------------------------+

Id:                  rh6-epel-live
Display Name:        rh6-epel-live
Description:         None
Content Unit Counts: 
  Erratum:          2023
  Package Category: 3
  Package Group:    208
  Rpm:              8336
Notes:               

Id:                  uploads
Display Name:        uploads
Description:         None
Content Unit Counts: 
  Rpm: 4
Notes:               

Id:                  copy-epel
Display Name:        copy-epel
Description:         None
Content Unit Counts: 
  Rpm: 8336
Notes:               

Id:                  rhel6_3
Display Name:        rhel6_3
Description:         None
Content Unit Counts: 
Notes:               

Id:                  test
Display Name:        test
Description:         None
Content Unit Counts: 
Notes:               

Id:                  zoo
Display Name:        zoo
Description:         None
Content Unit Counts: 
  Erratum:          4
  Package Category: 1
  Package Group:    2
  Rpm:              32
Notes:               


[root@preethi-el6-pulp ~]#
Comment 7 Preethi Thomas 2013-04-08 11:59:01 EDT
Pulp 2.1 released 


http://www.pulpproject.org/2013/04/05/pulp-2-1-0-released/

Note You need to log in before you can comment on or make changes to this bug.