Bug 590743

Summary: Add .xz support for, at least, .sqlite MD ... mostly solves the N versions of kernel with 9k provides problem
Product: Red Hat Enterprise Linux 6 Reporter: James Antill <james.antill>
Component: yumAssignee: James Antill <james.antill>
Status: CLOSED WONTFIX QA Contact: BaseOS QE Security Team <qe-baseos-security>
Severity: medium Docs Contact:
Priority: low    
Version: 6.0CC: jcm, jhutar
Target Milestone: rcKeywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-26 19:33:37 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description James Antill 2010-05-10 15:27:19 UTC 
Description of problem:

 This is kind of an RFE, however with kernel doing 9k provides this could be regarded as more of a bug. See SuSE's Bug about the problem:

https://bugzilla.novell.com/show_bug.cgi?id=586960

 Here is the data I tested with:

% l /tmp/kerns-repo -1
total 168M
 21M kernel-2.6.32-19.el6.i686.rpm
 23M kernel-2.6.32-19.el6.x86_64.rpm
 20M kernel-2.6.32-22.el6.i686.rpm
 22M kernel-2.6.32-22.el6.x86_64.rpm
 21M kernel-2.6.32-23.el6.i686.rpm
 22M kernel-2.6.32-23.el6.x86_64.rpm
 21M kernel-2.6.32-24.el6.i686.rpm
 22M kernel-2.6.32-24.el6.x86_64.rpm

...and an example of the data:

% rpm -qp --provides /tmp/kerns-repo/kernel-2.6.32-19.el6.i686.rpm | wc -l
9313

...then the createrepo data:

% l /tmp/kerns-repo/repodata -1
total 17M
 88K filelists.sqlite.bz2
108K filelists.xml.gz
 36K other.sqlite.bz2
100K other.xml.gz
1.7M primary.sqlite.bz2
960K primary.xml.gz

...and finally the primary.sqlite file both uncompressed and then re-compressed using "xz":

7.9M primary.sqlite
1.1M primary.sqlite.xz

...next I did:

% mkdir /tmp/kerns-repo/1
% mkdir /tmp/kerns-repo/2
% mkdir /tmp/kerns-repo/3
% mv /tmp/kerns-repo/*.rpm /tmp/kerns-repo/1
% ln /tmp/kerns-repo/1/* /tmp/kerns-repo/2
% ln /tmp/kerns-repo/1/* /tmp/kerns-repo/3

...which is a hacky way to get 24 versions of the kernel, instead of 8. Primary sizes again:

 24M primary.sqlite
4.7M primary.sqlite.bz2
2.0M primary.sqlite.xz

...and "time" results:

% time bzip2 -dc /tmp/kerns-repo/repodata/primary.sqlite.bz2 > /tmp/kerns-repo/repodata/primary.sqlite
bzip2 -dc /tmp/kerns-repo/repodata/primary.sqlite.bz2 >   1.19s user 0.13s system 97% cpu 1.352 total
% time xz -9 /tmp/kerns-repo/repodata/primary.sqlite
xz -9 /tmp/kerns-repo/repodata/primary.sqlite  21.57s user 0.57s system 95% cpu 23.145 total
% time xz /tmp/kerns-repo/repodata/primary.sqlite
xz /tmp/kerns-repo/repodata/primary.sqlite  20.96s user 0.24s system 96% cpu 21.945 total
% time unxz /tmp/kerns-repo/repodata/primary.sqlite.xz
unxz /tmp/kerns-repo/repodata/primary.sqlite.xz  0.45s user 0.11s system 95% cpu 0.584 total

...note that "xz -9" and "xz" both produced a 2MB primary.sqlite.

Version-Release number of selected component (if applicable):
3.2.27-7

 However note that this would just be the client side change, we'd also need a createrepo change (and an RHN change) to create the .sqlite.xz files.
Comment 1 James Antill 2010-05-10 15:31:46 UTC 
 Also, in case it wasn't obvious, yes if we _have_ to we can probably delay this until 6.1 _however_ that means anyone installing GA for all of RHEL-6 is going to download a bigger .sqlite.bz2 (and/or anyone who refuses to update yum).
 Where if we do the yum change for 6.0, then even if we don't do the server change until later all users will get the benefit.
Comment 2 RHEL Program Management 2010-05-10 15:40:40 UTC 
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 4 RHEL Program Management 2010-05-14 14:32:49 UTC 
Quality Engineering Management has reviewed and declined this request.  You may
appeal this decision by reopening this request.
Comment 5 James Antill 2010-06-23 16:50:01 UTC 
Moving to 6.1 ... note that we have the bindings now, so we just need the minor createrepo and yum changes to use it.
Comment 6 James Antill 2010-06-23 17:21:56 UTC 
Removing the bindings blocker
Comment 8 James Antill 2010-11-22 20:15:46 UTC 
moving to 6.2.x ... it's not done upstream, yet. And pyliblzma is only just available in Fed-14.
Comment 10 James Antill 2011-06-09 18:30:54 UTC 
Still not done upstream ... need to think about again, WRT repodata.
Comment 12 James Antill 2011-12-20 19:25:26 UTC 
 Given we opened it, and it's not on in Fedora, and there have been a couple of weird issues when people have tried it outside Fedora ... and that we'd still need to change the server side. I think we can just drop this for RHEL-6.
Comment 14 Suzanne Logcher 2012-05-18 20:45:40 UTC 
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Comment 15 James Antill 2012-06-26 19:33:37 UTC 
 Not much has changed since comment #12, closing this out.