Bug 1275345 - docker doesn't understand 'x-gzip' encoding and thus fails to pull images
Summary: docker doesn't understand 'x-gzip' encoding and thus fails to pull images
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker
Version: 7.2
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: rc
: ---
Assignee: Antonio Murdaca
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-10-26 15:08 UTC by Tomas Tomecek
Modified: 2019-03-01 10:21 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-01 10:21:05 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Tomas Tomecek 2015-10-26 15:08:47 UTC
If we have a docker v2 registry service (not distribution but pulp) behind httpd, httpd sends blobs with header specifying "x-gzip". Unfortunately docker engine doesn't understands this and fails to verify the blobs:

msg="Trying to pull registry:5000/pulptest/registry from https://registry:5000 v2"
msg="Pulling tag from V2 registry: \"latest\""
msg="Verification failed for /image using key Z36W:PUJT:F5GU:LGFM:NSSJ:K7JR:YVJ3:SUJY:DJTZ:PHWB:MSQ7:PZPU"
msg="Key check result: not verified"
msg="pulling blob \"sha256:bbe1c4256df30171585344c50fab278157cbf2cb3a2016bb720e99e49a759743\" to d3a1f33e8a5a513092f01bb7eb1c2abf4d711e5105390a3fe1ae2248cfde1391"
msg="pulling blob \"sha256:911d09728ffd9388b33c62f8ea09d751350a882f719794acb3b186dcf5b958df\" to c22013c8472965aa5b62559f2b540cd440716ef149756e7b958a1b2aba421e87"
msg="pulling blob \"sha256:615765bc0d9f82db061f7575d01bd94c99465a6e8dc1a45b793e6fd179e1ddae\" to d74508fb6632491cea586a1fd7d748dfc5274cd6fdfedee309ecdcbc2bf5cb82"
msg="pulling blob \"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\" to 91e54dfb11794fad694460162bf0cb0a4fa710cfa3f60979c177d920813e267c"
msg="pulling blob \"sha256:d50f4af10145cce25d286408daf9e38e47d5ad4c9363eb1da8f1e6b0a0fb6b27\" to bbb5ece9cf825bf5070ed61a30e3bd6eee1de27e385760ed8738c47c8e4ca0df"
msg="pulling blob \"sha256:c1a7c6af40aa2dd904c128fa58f1bcbf01689e879dac852e79ae3d412772eef4\" to 6b38cd1a3cd22139d65bf5d389e630c2c2e3b6d890960d4959406395690a053b"
msg="pulling blob \"sha256:a2416c2d43739818a86c548724967100ec83a8d89808aee3dae5b1a650a544aa\" to ae42fe1766c80192ab47749deace98f4869de71807372c1b3e2f01e80e8f4924"
msg="pulling blob \"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\" to 2be15b967cb0d2cc434c5c8e9817218acb50264cee056fdd0fea776bf7212033"
msg="pulling blob \"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\" to 1ff16d9d7c1d2dbf5c29a13ccf1f69e0f573035ed71269b1b49987565b9cd588"
msg="pulling blob \"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\" to f2bedebc25ccf9a8b3edda7defdbaffc14c8d1d10edfc6ba1aa3afce61052009"
msg="pulling blob \"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\" to cfb13e1d578049df78890fc0dfb2c926465a552d9112ec3c896c42cb1ff5d275"
msg="pulling blob \"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\" to cfb13e1d578049df78890fc0dfb2c926465a552d9112ec3c896c42cb1ff5d275"
msg="filesystem layer verification failed for digest sha256:a2416c2d43739818a86c548724967100ec83a8d89808aee3dae5b1a650a544aa"
msg="filesystem layer verification failed for digest sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
msg="filesystem layer verification failed for digest sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
msg="filesystem layer verification failed for digest sha256:911d09728ffd9388b33c62f8ea09d751350a882f719794acb3b186dcf5b958df"
msg="filesystem layer verification failed for digest sha256:615765bc0d9f82db061f7575d01bd94c99465a6e8dc1a45b793e6fd179e1ddae"
msg="filesystem layer verification failed for digest sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
msg="filesystem layer verification failed for digest sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
msg="filesystem layer verification failed for digest sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
msg="filesystem layer verification failed for digest sha256:c1a7c6af40aa2dd904c128fa58f1bcbf01689e879dac852e79ae3d412772eef4"
msg="filesystem layer verification failed for digest sha256:d50f4af10145cce25d286408daf9e38e47d5ad4c9363eb1da8f1e6b0a0fb6b27"
msg="filesystem layer verification failed for digest sha256:bbe1c4256df30171585344c50fab278157cbf2cb3a2016bb720e99e49a759743"
msg="Error trying v2 registry: filesystem layer verification failed for digest sha256:bbe1c4256df30171585344c50fab278157cbf2cb3a2016bb720e99e49a759743"
msg="error copying from layer download progress reader: download canceled"


Relevant line from /etc/httpd/conf/magic
0       string          \037\213        application/octet-stream        x-gzip


With curl, this works just fine and one can download and verify all blobs.


Version-Release number of selected component (if applicable):

docker-1.9.0-5.git11b81f9.fc24.x86_64

it also fails with 1.8 docker

Comment 1 Tomas Tomecek 2015-10-26 15:13:44 UTC
This is the response from server:

HTTP/1.1 200 OK
Date: Mon, 26 Oct 2015 13:30:22 GMT
Server: Apache/2.2.15 (Red Hat)
Last-Modified: Fri, 23 Oct 2015 12:13:20 GMT
ETag: \"43c38-11736-522c48997e618\"
Accept-Ranges: bytes
Content-Length: 71478
Docker-Distribution-API-Version: registry/2.0
Connection: close
Content-Type: application/x-tar
Content-Encoding: x-gzip


When getting rid of the "x-gzip", docker succeeds to pull the image.

Comment 2 smahajan@redhat.com 2015-11-01 18:50:56 UTC
Issue opened upstream.
https://github.com/docker/docker/issues/17595

Shishir

Comment 3 Daniel Walsh 2015-12-01 21:40:49 UTC
Any movement on this?

Comment 4 Tomas Tomecek 2016-01-07 12:12:41 UTC
Since docker upstream closed this issue, can we easily configure httpd not to populate http response with "Content-Encoding: x-gzip"?

Comment 6 Daniel Walsh 2016-01-07 15:00:38 UTC
So is this a bug on a configuration issue?

Comment 7 Jan Kaluža 2016-01-08 07:12:57 UTC
I admit I kind of don't like what mod_mime_magic does here and I will try to find out if it's OK to do Content-Encoding changes like that even in year 2016. The true is that Content-encoding definition allows that:

> The Content-Encoding entity-header field is used as a modifier to the
> media-type. When present, its value indicates what additional content codings
> have been applied to the entity-body, and thus what decoding mechanisms must
> be applied in order to obtain the media-type referenced by the Content-Type
> header field.

mod_mime_magic basically says here that there's tar file compressed by zip, which is right. So if you uncompress the httpd body, you will get tarball. It's probably up to the client to understand that it's actually tgz.

But as I said, I will try to discuss that with upstream developers to find out if that's really good thing to do.

Comment 8 Jan Kaluža 2016-01-08 10:12:36 UTC
I've started discussion on httpd-dev if you want to follow up. I'm on PTO next week, so I will check the end of that discussion after that.

http://mail-archives.apache.org/mod_mbox/httpd-dev/201601.mbox/%3C568F6A11.1060309%40redhat.com%3E

Comment 9 Tomas Tomecek 2016-01-08 12:02:50 UTC
(In reply to Daniel Walsh from comment #6)
> So is this a bug on a configuration issue?

I hope this is just a configuration issue.

(In reply to Jan Kaluža from comment #7)
> I admit I kind of don't like what mod_mime_magic does here and I will try to
> find out if it's OK to do Content-Encoding changes like that even in year
> 2016. The true is that Content-encoding definition allows that:
> 
> > The Content-Encoding entity-header field is used as a modifier to the
> > media-type. When present, its value indicates what additional content codings
> > have been applied to the entity-body, and thus what decoding mechanisms must
> > be applied in order to obtain the media-type referenced by the Content-Type
> > header field.
> 
> mod_mime_magic basically says here that there's tar file compressed by zip,
> which is right. So if you uncompress the httpd body, you will get tarball.
> It's probably up to the client to understand that it's actually tgz.
> 
> But as I said, I will try to discuss that with upstream developers to find
> out if that's really good thing to do.

I got to admit that I agree with docker upstream here. The compression here is not used to save bandwidth, nor it was done by httpd, the compression was used due to application design and the application is requiring it. Therefore I consider it being a bug that httpd is setting a Content-Encoding header.

Comment 10 Joe Orton 2016-01-11 11:50:18 UTC
I added comments from here:

https://github.com/docker/docker/issues/17595#issuecomment-169976857

I don't see any good argument that there is a server bug here, but I agree that the configuration should be fixed to not rely on mod_mime_magic for MIME type sniffing regardless, which would work around the bad client behaviour.

This is the kind of issue that really should be fixed in the client if you anybody upstream is serious about using a RESTful protocol, rather than having to tweak server configs when you hit an interoperability issue.

Comment 11 Petr Pisar 2016-01-11 12:46:23 UTC
I recommend to docker client to explicitly disable content-encodings other than "identity" by sending empty "Accept-Encoding:" header. See <http://tools.ietf.org/html/rfc2616#section-14.3>:

         [...] If the
         Accept-Encoding field-value is empty, then only the "identity"
         encoding is acceptable.

Comment 12 Tomas Tomecek 2016-01-11 14:50:31 UTC
Would it be possible to open a pull request to fix this by "disabling" the encoding? Looks like docker could even merge it if they suggest it: https://github.com/docker/docker/issues/17595#issuecomment-170122669

(In reply to Petr Pisar from comment #11)
> I recommend to docker client to explicitly disable content-encodings other
> than "identity" by sending empty "Accept-Encoding:" header. See
> <http://tools.ietf.org/html/rfc2616#section-14.3>:
> 
>          [...] If the
>          Accept-Encoding field-value is empty, then only the "identity"
>          encoding is acceptable.

Thanks for help.

Comment 13 Jan Kaluža 2016-01-20 14:14:14 UTC
I just want to share the overall impression from httpd-dev thread about this issue. It looks like the consensus is that there should be a way to disable this behaviour in a better way than disabling mod_mime_magic completely (which is the only way how to disable this httpd behaviour currently).

However, we won't be able to disable that globally in default configuration in the RHEL7, because the change is not backward compatible.

So even with a way how to disable this behaviour on httpd side, users would have to disable that manually and therefore I think it's up to Docker devs (or the documentation) to address this issue somehow.

Comment 14 Daniel Walsh 2016-02-22 19:22:38 UTC
Shishir was an update request ever made to docker-distribution on this?

Comment 15 smahajan@redhat.com 2016-03-30 20:02:48 UTC
This seems to be a change to docker-distribution.
Can we reassign this bugzilla to the distribution team ?

Shishir

Comment 16 Michal Minar 2016-10-06 16:29:39 UTC
This is unrelated to docker-distribution. The original problem mentions pulp, not docker-distribution. The problem is in a configuration of httpd server in front of the pulp.

Tomas, what's the right component here? It seems like you run pulp locally. Is it a package built for Fedora or RHEL?

Comment 17 Tomas Tomecek 2016-10-10 07:33:59 UTC
I opened this against docker, not docker-distribution. Anyway, we had this problem one year ago and I'm not sure if this is still relevant.

Tim, can you clarify please?

Comment 18 Tim Waugh 2016-10-10 09:57:33 UTC
I think this is related to an upstream issue:
  https://github.com/docker/docker/pull/25122#issuecomment-245600858

which is awaiting information from me I have not had the time to provide.

Comment 19 Michal Minar 2016-10-13 14:03:53 UTC
OK, so the fix is related to docker daemon. Changing the component again. It seems like a low priority. Thanks all for the pointers.

Comment 20 Antonio Murdaca 2016-10-23 14:09:21 UTC
Michal, this was a docker/distribution issue since the client used to fetch layers lives into docker/distribution.

Anyhow, based on upstream discussion, I've opened https://github.com/docker/distribution/pull/2015 to solve https://github.com/docker/docker/issues/27642.

Did anyone test that an "Accept: identity" header solve this issue though? I can test it out if someone provides me with a consistent reproducer (I don't know how to setup a pulp repo for docker)

Comment 21 Tim Waugh 2016-10-24 10:11:58 UTC
I haven't tested this yet. I don't think it needs a Pulp repository, only httpd serving the same responses a registry would, but configured to set the Content-Type header from the content of the file (which I understand is the default anyway).

Comment 22 Daniel Walsh 2017-06-30 14:59:49 UTC
I have no idea if this is still an issue or it belongs to docker or docker-distribution?

Comment 23 Tomas Tomecek 2019-03-01 10:21:05 UTC
I no longer care about this, hence closing.


Note You need to log in before you can comment on or make changes to this bug.