Bug 1532233 - wget incorrectly decompresses downloads
Summary: wget incorrectly decompresses downloads
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: wget
Version: 27
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Tomáš Hozza
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-08 12:34 UTC by Wolfgang Denk
Modified: 2018-02-06 10:48 UTC (History)
2 users (show)

Fixed In Version: wget-1.19.4-1.fc27 wget-1.19.4-1.fc26
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-01-30 18:05:06 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Wolfgang Denk 2018-01-08 12:34:39 UTC
Description of problem:

Starting with version v1.19.x, wgeht has support for a new option:

--compression=type
           Choose the type of compression to be used.  Legal values are auto, gzip and none.

           If auto or gzip are specified, Wget asks the server to compress the file using the gzip
           compression format. If the server compresses the file and responds with the "Content-Encoding"
           header field set appropriately, the file will be decompressed automatically. This is the
           default.

This default setting causes that downloads get uncompressed on the fly, which breaks zillions of build scripts, include contless Yocto / OE recipes.


Version-Release number of selected component (if applicable):

wget-1.19.2-2.fc27.x86_64

How reproducible:

always

Steps to Reproduce:
1. Run:

$ wget http://www.lysator.liu.se/~nisse/archive/nettle-3.3.tar.gz
--2018-01-08 13:28:08--  http://www.lysator.liu.se/~nisse/archive/nettle-3.3.tar.gz
Resolving www.lysator.liu.se (www.lysator.liu.se)... 130.236.254.11, 2001:6b0:17:f0a0::b
Connecting to www.lysator.liu.se (www.lysator.liu.se)|130.236.254.11|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1887927 (1.8M) [application/unix-tar]
Saving to: ‘nettle-3.3.tar.gz’

nettle-3.3.tar.gz           100%[=========================================>]   1.80M  4.00MB/s    in 0.4s    

2018-01-08 13:28:09 (4.00 MB/s) - ‘nettle-3.3.tar.gz’ saved [6184960]

You can see from the "Length: 1887927 (1.8M)" entry that the actual (compressed) file length is 1.8 MB, but the file sored on disk is 6184960 bytes = 5.9 MB.
Also, despite of the name, it is no ".tar.gz" but a plain tar file:

$ file nettle-3.3.tar.gz
nettle-3.3.tar.gz: POSIX tar archive (GNU)

Actual results:

Files gets uncompressed on the fly.  All integrity tests (size, checksum) will fail. Attempts to unzip the expected gzipped file will fail, too.

This breaks compatibility with a zillion of build scripts.

Expected results:

File downloaded without any modifications, with correct size and checksum.

Additional info:

For some reason, in the example the download works fine when using HTTPS instead of HTTP (but this is probably a specific feature/(mis)configuration of the server).

The problem can be worked around by adding
    compression = none
to /etc/wgetrc or the user's local .wgetrc

Note that this is only a workaround - the breakage must be fixed with the default settings.

Comment 1 Tomáš Hozza 2018-01-08 12:53:13 UTC
Hello.

Thank you for the report. This is a known issue to the upstream (http://lists.gnu.org/archive/html/bug-wget/2017-11/msg00000.html). The workaround is exactly the same as you described in the bug description (to set "compression = none" in wgetrc).

However there is still no fix in the upstream git. I'll follow up with them to see if we can fix this in Fedora soon...

Comment 2 Fedora Update System 2018-01-22 15:03:40 UTC
wget-1.19.4-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-83fc15f299

Comment 3 Fedora Update System 2018-01-22 15:03:50 UTC
wget-1.19.4-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2018-a2eebd7c4c

Comment 4 Fedora Update System 2018-01-23 22:49:41 UTC
wget-1.19.4-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-a2eebd7c4c

Comment 5 Fedora Update System 2018-01-23 23:41:27 UTC
wget-1.19.4-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-83fc15f299

Comment 6 Fedora Update System 2018-01-30 18:05:06 UTC
wget-1.19.4-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 7 Fedora Update System 2018-02-06 10:48:05 UTC
wget-1.19.4-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.