Bug 441110 - RPM LZMA payload
RPM LZMA payload
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: distribution (Show other bugs)
rawhide
All Linux
low Severity low
: ---
: ---
Assigned To: Bill Nottingham
Bill Nottingham
: FutureFeature
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-04-06 07:13 EDT by Rahul Sundaram
Modified: 2014-03-16 23:13 EDT (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-08-10 02:41:30 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
suse-10.3-x86_64 mock setup (6.47 KB, application/x-download)
2009-08-10 12:24 EDT, Ralf Corsepius
no flags Details

  None (edit)
Description Rahul Sundaram 2008-04-06 07:13:46 EDT
Description of problem:

SUSE is apparently moving to LZMA for their next version and have published some
benchmarks

http://www.kdedevelopers.org/node/3326

Mandriva already uses LZMA. 

RPM discussion

http://www.mail-archive.com/rpm-maint@lists.rpm.org/msg00547.html
Comment 1 Jeremy Katz 2008-04-06 22:24:52 EDT
And has there been an attempt to actually get these changes into upstream RPM?
Comment 2 Rahul Sundaram 2008-04-07 09:45:24 EDT
AFAIK, this is now upstream already. Refer

https://lists.dulug.duke.edu/pipermail/rpm-maint/2008-April/000872.html

Comment 3 Jeremy Katz 2008-04-07 09:50:59 EDT
That's lzma compressed sources (ie Source0: foo.tar.lzma or whatnot), not lzma
payloads.
Comment 4 Panu Matilainen 2008-04-08 01:43:41 EDT
Support for LZMA-payloads is certainly going to go into upstream sooner or
later, but it's not there yet. I'm waiting for the next major version of
lzma-utils with a usable library to be released, I'm not too keen on carrying an
lzma-decoder within rpm (which is how Mandriva does it)
Comment 5 Panu Matilainen 2008-04-15 07:20:41 EDT
Suse seems to be doing the right thing (imo) here: they've added the lzma
payload support by developing against current alpha-version of liblzma. The
patch looks fairly complete and sane, I'm expecting to forward-port it to
rpm.org HEAD soonish.
Comment 6 Panu Matilainen 2008-05-08 02:55:22 EDT
Support for LZMA payloads is now in rpm.org HEAD, forward-ported by Jindrich.
Comment 7 Bug Zapper 2008-05-14 04:58:13 EDT
Changing version to '9' as part of upcoming Fedora 9 GA.
More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 8 Kevin Kofler 2008-06-19 17:36:19 EDT
Back to Rawhide.
Comment 9 Amadeus 2008-06-21 12:22:12 EDT
Compression and decompressing scales almost perfectly on multi core cpu's, so
perhaps it would make sense to first make liblzma multi threaded before changing
to lzma for rpm?

Comment 10 Panu Matilainen 2008-07-14 08:53:59 EDT
The new rpm in rawhide has support for LZMA payloads, but only when built
against lzma-utils >= lzma-4.42.2alpha, whereas rawhide is still on 4.32.x which
is the stable lzma branch.

From this point on, enabling lzma compression of payloads is a distro, not
rpm-level, if and when question (alpha release/snapshot of lzma libs could be
separately packaged for rpm needs, which is what SuSE did).
Comment 11 Bill Nottingham 2008-07-14 11:39:31 EDT
FWIW, I don't see this as being useful to add until any release that we support
upgrading from also supports LZMA payloads.

Panu - how backportable is the support?
Comment 12 Ralf Corsepius 2008-07-14 13:00:17 EDT
(In reply to comment #11)
> FWIW, I don't see this as being useful to add until any release that we support
> upgrading from also supports LZMA payloads.
FWIW: Fedora's rpm lacking lzma payloads disqualifies Fedora's mock from being
usable as a build host for SUSE-packages.
Comment 13 Panu Matilainen 2008-07-16 05:00:50 EDT
The LZMA payload support doesn't need much in the way of backporting as SuSE
developed it on top of their patched 4.4.2 rpm and we forward-ported in to
rpm.org HEAD. 

Even so, my suggestion would be considering moving to LZMA payloads when F11
development starts:
- With a little bit of luck, a stable version of new lzma utils might be out by
then. If not, we need to consider if we want to base such a critical piece on an
unreleased version.
- LZMA support for F10 rpm can be easily enabled at any time by just rebuilding,
only depends on the above case. So the reasonable upgrade path would be there,
or at least could be easily arranged.
Comment 14 Ralf Corsepius 2008-11-11 08:59:05 EST
Sad, but true. apparently, not much progress on this front.

FC10's rpm, yum and mock are as unusable for building suse-packages as FC9's rpm, yum and mock had been.
Comment 15 Panu Matilainen 2008-11-12 06:50:08 EST
Suse made a gamble with unstable lzma-utils and they're going to pay for it. lzma-utils upstream just recently changed the file format (again), and it wont even read the files created by the intermediate alpha version Suse uses:

[pmatilai@localhost tmp]$ lzma --version|head -1
lzma (LZMA Utils) 4.42.3alpha
[pmatilai@localhost tmp]$ lzma rpm-4.6.0-rc1.tar 
[pmatilai@localhost tmp]$ ~/repos/lzma-utils/src/lzma/lzma --version|head -1
lzma (LZMA Utils) 4.999.6alpha
[pmatilai@localhost tmp]$ ~/repos/lzma-utils/src/lzma/lzma --test rpm-4.6.0-rc1.tar.lzma 
/home/pmatilai/repos/lzma-utils/src/lzma/lzma: rpm-4.6.0-rc1.tar.lzma: File format not recognized

Better wait until a stable release...
Comment 16 Bug Zapper 2008-11-25 21:11:39 EST
This bug appears to have been reported against 'rawhide' during the Fedora 10 development cycle.
Changing version to '10'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 17 Rahul Sundaram 2009-06-13 04:07:52 EDT
setting it back
Comment 18 Ralf Corsepius 2009-07-30 01:36:29 EDT
It seems the xz-enabled RH-rpm is able process SUSE rpms with lzma payload.

No idea yet, if it processes them correctly.
Comment 19 Panu Matilainen 2009-08-10 02:41:30 EDT
(In reply to comment #18)
> It seems the xz-enabled RH-rpm is able process SUSE rpms with lzma payload.
> 
> No idea yet, if it processes them correctly.  

It's supposed to be compatible with Suse (and Mandriva), both of which use the "old" lzma-alone format for payload compression. File separate bugs if you find issues.

All of rawhide has been built with XZ (LZMA successor format) payload compression now, considering this done.
Comment 20 Ralf Corsepius 2009-08-10 09:41:25 EDT
(In reply to comment #19)
> (In reply to comment #18)
> > It seems the xz-enabled RH-rpm is able process SUSE rpms with lzma payload.
> > 
> > No idea yet, if it processes them correctly.  
> 
> It's supposed to be compatible with Suse (and Mandriva), both of which use the
> "old" lzma-alone format for payload compression. File separate bugs if you find
> issues.
Well, I am facing bizarre dependency issues when trying to use suse repositories.

Either their EVR comparison operations work entirely different than Red Hat's, their repos are incompatible to Red Hat's yum or their yum repos are entirely messed up.

E.g.
mock -r suse-11.0-x86_64 init
...
 # /usr/bin/yum --installroot /var/lib/mock/suse-11.0-x86_64/root/  update
perl-5.10.0-37.8.x86_64 from installed has depsolving problems
  --> Missing Dependency: perl-base = 5.10.0 is needed by package perl-5.10.0-37.8.x86_64 (installed)
 You could try using --skip-broken to work around the problem
 You could try running: package-cleanup --problems
                        package-cleanup --dupes
                        rpm -Va --nofiles --nodigest
The program package-cleanup is found in the yum-utils package.
Error: Missing Dependency: perl-base = 5.10.0 is needed by package perl-5.10.0-37.8.x86_64 (installed)


# rpm -q --provides -p /var/cache/mock/suse-11.1-x86_64/yum_cache/base/packages/perl-base-5.10.0-62.16.x86_64.rpm | grep perl-base
perl-base = 5.10.0-62.16

# rpm -q --requires -p /var/cache/mock/suse-11.1-x86_64/yum_cache/update/packages/perl-5.10.0-62.18.1.x86_64.rpm | grep perl-base
perl-base = 5.10.0

So, I am inclined to think suse might be treating 
Requires: %version as %version >= %version-"" (release=empty),
while Red Hat seems to be treating 
%version as exact string match.
Comment 21 Michael Schwendt 2009-08-10 10:15:20 EDT
> So, I am inclined to think suse might be treating 
> Requires: %version as %version >= %version-"" (release=empty),

There, >= would make no sense if the spec says '='.

RPM in Red Hat/Fedora satisfies "Requires: %name = %version" with exact %version but any %release. 

[Further, since %release is not part of %version, something like 5.10.0-62.16 is not "higher than" 5.10.0 - but that's a mistake only done by packaging newbies]
Comment 22 Ralf Corsepius 2009-08-10 10:29:21 EDT
(In reply to comment #21)
> RPM in Red Hat/Fedora satisfies "Requires: %name = %version" with exact
> %version but any %release. 
I am observing exactly the converse:

Red Hat rpm/yum/mock fails with dependency resolution issue on SuSE repos because SUSE-rpms provide "V-R", while some of their packages require "V" (without %release)

Example (here suse-11.1):

# rpm -q --provides -p /var/cache/mock/suse-11.1-x86_64-rtems/yum_cache/base/packages/perl-base-5.10.0-62.16.x86_64.rpm | grep perl-base
perl-base = 5.10.0-62.16

# rpm -q --requires -p /var/cache/mock/suse-11.1-x86_64-rtems/yum_cache/update/packages/perl-5.10.0-62.18.1.x86_64.rpm | grep perl-base
perl-base = 5.10.0

Fedora 11's mock/yum/rpm is not able to resolve this issue and fails.

FWI: I am also observing that the number of such issue increase with SUSE releases, i.e. suse-10.3 has less such issues than 11.0, and 11.0 less such issues than 11.1 (11.1 under Fedora 11 mock/rpm/yum is 1st class disaster).
Comment 23 Michael Schwendt 2009-08-10 11:31:06 EDT
> Example (here suse-11.1):

That example only demonstrates that their RPM version comparison is expected to work exactly like Fedora's. [Or else they could not require "perl-base = 5.10.0" and expect perl-base = 5.10.0-62.16 to satisfy the dep.]


> Fedora 11's mock/yum/rpm is not able to resolve this issue and fails.

So, can you create a reduced [or minimal] test-case to show that?

Does F11's RPM alone fail for you when trying to install exemplary openSUSE packages with such dependencies?

Or is it the combination of yum+rpm or just mock+yum+rpm? Or perhaps it's even important for the packages to be built with suse's RPM?

[...]

I can show that F11's RPM handles such dependencies as expected (they've come up during broken deps and bad Obsoletes/Provides reports regularly):

$ sudo rpm -ivh subpkg-1.0-2.fc11.i586.rpm 
error: Failed dependencies:
        goodpkg = 1.0 is needed by subpkg-1.0-2.fc11.i586
        something = 2.0 is needed by subpkg-1.0-2.fc11.i586

$ rpm -qp --provides goodpkg-1.0-2.fc11.i586.rpm 
something = 2.0-4
goodpkg = 1.0-2.fc11
goodpkg(x86-32) = 1.0-2.fc11

$ sudo rpm -ivh subpkg-1.0-2.fc11.i586.rpm goodpkg-1.0-2.fc11.i586.rpm 
Preparing...                ########################################### [100%]
   1:goodpkg                ########################################### [ 50%]
   2:subpkg                 ########################################### [100%]
Comment 24 Ralf Corsepius 2009-08-10 12:24:39 EDT
Created attachment 356918 [details]
suse-10.3-x86_64 mock setup
Comment 25 Ralf Corsepius 2009-08-10 12:34:56 EDT
(In reply to comment #23)

> > Fedora 11's mock/yum/rpm is not able to resolve this issue and fails.
> 
> So, can you create a reduced [or minimal] test-case to show that?

Yes. Install the tarball above on Fedora 11/x86_64:
cd /; tar xjvf suse-mock-demo.tar.bz2

Then try the following:
# mock -r suse-10.3-x86_64 --init
# mock -r suse-10.3-x86_64 --update

Result:

# mock -r suse-10.3-x86_64 --update
INFO: mock.py version 0.9.16 starting...
State Changed: init plugins
State Changed: start
ERROR: Command failed: 
 # /usr/bin/yum --installroot /var/lib/mock/suse-10.3-x86_64/root/  update
perl-5.8.8-76.4.x86_64 from installed has depsolving problems
  --> Missing Dependency: perl-base = 5.8.8 is needed by package perl-5.8.8-76.4.x86_64 (installed)
Error: Missing Dependency: perl-base = 5.8.8 is needed by package perl-5.8.8-76.4.x86_64 (installed)
 You could try using --skip-broken to work around the problem
 You could try running: package-cleanup --problems
                        package-cleanup --dupes
                        rpm -Va --nofiles --nodigest
The program package-cleanup is found in the yum-utils package.


# rpm -q --provides -p /var/cache/mock/suse-10.3-x86_64/yum_cache/base/packages/perl-base-5.8.8-76.x86_64.rpm  | grep perl-base
perl-base = 5.8.8-76

# rpm -q --requires -p /var/cache/mock/suse-10.3-x86_64/yum_cache/update/packages/perl-5.8.8-76.4.x86_64.rpm | grep perl-base
perl-base = 5.8.8

> Does F11's RPM alone fail for you when trying to install exemplary openSUSE
> packages with such dependencies?
How to test this? You normally can't install openSUSE rpms on RH for various reasons (Fedora's rpm lacking lzma compression until recently was one "big road block").

> Or is it the combination of yum+rpm or just mock+yum+rpm?
My test case is "building rpms targetting for openSUSE using Fedora 11's mock"

> Or perhaps it's even
> important for the packages to be built with suse's RPM?
The issues I am referring to are with openSUSE packages during setting up a mock environment, i.e. mostly only genuine openSUSE packages are involved.
Comment 26 Michael Schwendt 2009-08-10 14:42:50 EDT
Unrelated to the payload.

We should move this elsewhere. fedora-devel-list e.g. or Yum upstream.

openSUSE 10.3 metadata is pretty much incompatible here with F11 also for i586. And it smells a lot as if the old "no Epoch => zero Epoch" issue is involved. For example, I see dependency rpm:entry tags without "epoch" attributes. The missing Epoch ("None" in Python) reveals a bug in repoclosure ( http://yum.baseurl.org/ticket/193 ) which causes %version to be missing in versioned deps. And even if fixed, I get tons of broken deps, such as "python >= 0:2.5" or "splashy = 0.3.3" while splashy-0.3.3-45.2 clearly is in the repo and provides the default "splashy = 0.3.3-45.2". But still, Yum whatProvides does not resolve it. It wouldn't surprise me if transparently added Epoch 0 (somewhere) causes all this.
Comment 27 Panu Matilainen 2009-08-11 02:14:25 EDT
Yup this doesn't belong here... but just FWIW, rpm itself finds no problems in the package set (whereas I can easily reproduce the "mock --update" failure):
[root@localhost ~]# rpm -Va --nofiles --root /var/lib/mock/suse-10.3-x86_64/root/
[root@localhost ~]#
Comment 28 Ralf Corsepius 2009-08-11 02:53:09 EDT
(In reply to comment #27)
> Yup this doesn't belong here... 
... yes, meanwhile we know.

Michael seems to be right, it's suse's update repos' metadata which seems to be incompatible to RH's yum (RH's yum seems to be able to process the "distribution" repos).

Note You need to log in before you can comment on or make changes to this bug.