Bug 501636 - UPX is compiled without LZMA support
UPX is compiled without LZMA support
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: upx (Show other bugs)
12
All Linux
low Severity medium
: ---
: ---
Assigned To: Jon Ciesla
Fedora Extras Quality Assurance
: Reopened, Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-05-20 01:38 EDT by H. Peter Anvin
Modified: 2010-01-27 19:56 EST (History)
2 users (show)

See Also:
Fixed In Version: 3.04-2.fc12
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-01-27 19:56:06 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
new upx.spec (3.50 KB, text/plain)
2010-01-07 00:11 EST, John Reiser
no flags Details
upx-3.03-compress_lzma.patch (17.21 KB, patch)
2010-01-07 00:14 EST, John Reiser
no flags Details | Diff
upx-3.03-Makevars-lzma.patch (957 bytes, patch)
2010-01-07 00:14 EST, John Reiser
no flags Details | Diff
lzma465.tar.bz2 (281.37 KB, application/octet-stream)
2010-01-07 00:16 EST, John Reiser
no flags Details

  None (edit)
Description H. Peter Anvin 2009-05-20 01:38:36 EDT
Description of problem:

The Fedora build of UPX still excludes LZMA support.  UPX does substantially better (often making the difference between worth using and not) when LZMA is available.

Version-Release number of selected component (if applicable):

upx-3.03-1.fc10.x86_64

How reproducible:

100%

Steps to Reproduce:
1. upx --lzma executable
  
Actual results:

upx: packer_c.cpp:43: static bool Packer::isValidCompressionMethod(int): Assertion `0 && "Internal error - LZMA not compiled in"' failed.
Abort

Expected results:

Compression with LZMA

Additional info:
Comment 1 Jon Ciesla 2009-05-20 09:50:24 EDT
Not an easy thing to fix:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=452817

I'll ask upstream where we sit here.
Comment 2 Jon Ciesla 2009-10-01 11:51:31 EDT
Pinged upstream again.
Comment 3 Bug Zapper 2009-11-18 06:59:28 EST
This message is a reminder that Fedora 10 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 10.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '10'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 10's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 10 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 4 Bug Zapper 2009-12-18 04:28:03 EST
Fedora 10 changed to end-of-life (EOL) status on 2009-12-17. Fedora 10 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.
Comment 5 H. Peter Anvin 2009-12-29 13:15:35 EST
Still there in 12.  This crap is so bloody annoying.
Comment 6 John Reiser 2010-01-04 16:18:22 EST
(In reply to comment #1)
> Not an easy thing to fix:
> 
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=452817

UPX uses LZMA source files directly, with #defines and other compile-time magic in UPX that make it possible to access portions of LZMA (particularly its structs) in ways that the author of LZMA did not anticipate, without violating original LZMA licensing [1998 or so], which was either CPL or [paraphrasing] "no source changes allowed".  UPX does not, and cannot, use any separately-compiled LZMA library.  The reason why UPX does this is because UPX requires total control over resource usage during *de*compression (no malloc() allowed, etc.), and this in turn requires near-total control over resource allocation during compression.  Despite relaxed licensing (LGPL2.1 at least as far back as lzma443 in 2005) LZMA has not been interested in providing nor facilitating the necessary control, and this stance is unlikely to change.

Therefore the only reasonable solution to providing --lzma for UPX in Fedora, is by copying the required files from LZMA into UPX, and ajusting the pathnames in the #includes.

[I am an author of UPX, although not the one who setup the usage of LZMA within UPX.]
Comment 7 Jon Ciesla 2010-01-05 10:36:40 EST
(In reply to comment #6)
> (In reply to comment #1)
> > Not an easy thing to fix:
> > 
> > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=452817
> 
> UPX uses LZMA source files directly, with #defines and other compile-time magic
> in UPX that make it possible to access portions of LZMA (particularly its
> structs) in ways that the author of LZMA did not anticipate, without violating
> original LZMA licensing [1998 or so], which was either CPL or [paraphrasing]
> "no source changes allowed".  UPX does not, and cannot, use any
> separately-compiled LZMA library.  The reason why UPX does this is because UPX
> requires total control over resource usage during *de*compression (no malloc()
> allowed, etc.), and this in turn requires near-total control over resource
> allocation during compression.  Despite relaxed licensing (LGPL2.1 at least as
> far back as lzma443 in 2005) LZMA has not been interested in providing nor
> facilitating the necessary control, and this stance is unlikely to change.

So do I understand that this might impact efforts to switch UPX to using xz?

> Therefore the only reasonable solution to providing --lzma for UPX in Fedora,
> is by copying the required files from LZMA into UPX, and ajusting the pathnames
> in the #includes.

I experimented with that, with little success.  Have you been able to make this work?  If this can be done, I could see doing this in Fedora, though it's far from ideal.

> [I am an author of UPX, although not the one who setup the usage of LZMA within
> UPX.]
Comment 8 John Reiser 2010-01-05 11:34:23 EST
>> UPX uses LZMA source files directly, ...

> So do I understand that this might impact efforts to switch UPX to using xz?

Yes, definitely.  UPX wants to decompress by specifying the character array INput, the character array OUTput, and all temporary space as one  pre-allocated character array INput.  This is a reasonable expectation for the environment in which UPX decompression operates: completely standalone, no libraries of any kind.

>> Therefore the only reasonable solution ... is by copying the required files
>> from LZMA into UPX ...

> Have you been able to make this work?

The upx/src/Makefile does this "as if", using $UPX_LZMADIR and $UPX_UCLDIR.  "gcc -E -MM" gives the specific recipe, which UPX records in file ".depend".  Only one file is affected, compress_lzma.o.  As of lzma463 the list is:
-----
compress_lzma.o: compress_lzma.cpp conf.h version.h miniacc.h \
 $UPX_UCLDIR/include/ucl/uclconf.h $UPX_UCLDIR/include/ucl/ucl.h \
 snprintf.h stdcxx.h options.h except.h bele.h bele_policy.h util.h \
 console.h compress.h mem.h $UPX_LZMADIR/C/7zVersion.h \
 $UPX_LZMADIR/C/Types.h $UPX_LZMADIR/C/LzmaEnc.h \
 $UPX_LZMADIR/C/Types.h $UPX_LZMADIR/C/LzmaEnc.c \
 $UPX_LZMADIR/C/LzmaEnc.h $UPX_LZMADIR/C/LzFind.h \
 $UPX_LZMADIR/C/LzFind.c $UPX_LZMADIR/C/LzHash.h \
 $UPX_LZMADIR/C/LzmaDec.h $UPX_LZMADIR/C/LzmaDec.c \
 $UPX_LZMADIR/C/LzmaDec.h
-----
Note that LzmaEnc.c and LzmaDec.c and LzmaFind.c are #included in addition to several *.h files.  Therefore: create a subdirectory such as "lzma_src/C/", copy those 9 *.h and 3 *.c files, and adjust UPX_LZMADIR.  The upx/src/Makefile already has "INCLUDES += -I$(UPX_LZMADIR)", and a typical line in upx/src/compress_lzma.cpp is '#include "C/LzmaDec.h"'.
Comment 9 John Reiser 2010-01-07 00:11:36 EST
Created attachment 382149 [details]
new upx.spec

I'm attaching a new .spec, two patches, and a SOURCE file (result of wget) that work for me.  Probably you can revise this to be better.
Comment 10 John Reiser 2010-01-07 00:14:01 EST
Created attachment 382150 [details]
upx-3.03-compress_lzma.patch

fast-forward to current compress_lzma.cpp
Comment 11 John Reiser 2010-01-07 00:14:50 EST
Created attachment 382151 [details]
upx-3.03-Makevars-lzma.patch

fast-forward to current Makevars.lzma
Comment 13 Jon Ciesla 2010-01-08 10:09:02 EST
Awesome, thanks!  That works!  New test binary compressed using upx --lzma was just under %30 of the original size.

I'll get this in rawhide today and F-12-testing very soon.
Comment 14 Fedora Update System 2010-01-08 11:03:26 EST
upx-3.04-2.fc12 has been submitted as an update for Fedora 12.
http://admin.fedoraproject.org/updates/upx-3.04-2.fc12
Comment 15 Fedora Update System 2010-01-12 18:33:54 EST
upx-3.04-2.fc12 has been pushed to the Fedora 12 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update upx'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F12/FEDORA-2010-0404
Comment 16 Fedora Update System 2010-01-27 19:56:00 EST
upx-3.04-2.fc12 has been pushed to the Fedora 12 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.