Bug 1324150

Summary: [golang 1.6] Provide some mechanism to ignore invalid 'host' headers
Product: Red Hat Enterprise Linux 7 Reporter: Eric Paris <eparis>
Component: dockerAssignee: Daniel Walsh <dwalsh>
Status: CLOSED ERRATA QA Contact: atomic-bugs <atomic-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.3CC: amurdaca, ccoleman, dwalsh, jhonce, lsm5, lsu, mmcgrath
Target Milestone: rcKeywords: Extras
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-06-23 16:18:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Eric Paris 2016-04-05 16:12:39 UTC
https://github.com/docker/docker/issues/20865

In golang 1.6 the core http libraries start validating that the 'host' http header is valid (as per the http spec).

golang 1.5 I believe started to not setting the 'host' header.

I believe upstream docker built docker 1.9 with golang 1.4.  Docker 1.10 with 1.5

We have build all versions of docker with 1.4.

So all of our docker builds and all upstream <=1.9 are setting the host header to "/var/lib/docker/".

If we build docker with golang 1.6 no currently shipping docker clients will be able to talk to that server since the server will reject the request given the invalid header. Anyone who has built a container with docker inside which mounts the docker socket into the container will stop working when the docker daemon is updated.

I find that position unacceptable that updating the golang compiler and rebuilding docker will break lots of existing containers.

I propose that we implement a hack in the golang 1.6 http library to allow a program to selectively disable the host header validation check. I think that the hack should print a warning of some sort when an invalid host header is found on the server. I believe that our version of docker 1.10 should make use of this hack. After $PERIOD_OF_TIME I think our docker should stop using the hack and we stop supporting old clients. Where $PERIOD_OF_TIME is TBD.

After the golang changes are implemented (and pushed/proposed upstream) we need  to open a BZ against docker.

Comment 2 Antonio Murdaca 2016-04-11 09:11:25 UTC
I don't think we should hack golang - the real issue here is Docker because golang is just fixing an invalid implementation of http 1.1 where the host header needs validatation. That said, I understand this will break lots of stuff but I think the proper hack should just go into Docker. We could read and validate the Host header in the daemon and sanitize it if we assume that a Host header starting with / comes from older (built with 1.4) Docker clients. This should have little overhead for newer clients (just read the first 2 http protocol line) and just sanitize the header if it's *wrong*.

Comment 3 Eric Paris 2016-04-11 13:02:22 UTC
@antonio I'm all for such a solution. I like it much better than a custom hack on the core libs. My understanding was that there was no a way the server could get access to the header to sanitize it before the new validation check.

If that is not the case and this can be fixed entirely in the docker daemon then it should be done there. And doing so should be seen as a high priority issue as 1.6 makes its way out.

Moving component to docker if they can do it all themselves.

Comment 4 Daniel Walsh 2016-04-11 14:29:36 UTC
Antonio why hasn't docker done this themselves then?

Comment 5 Antonio Murdaca 2016-04-11 14:31:25 UTC
Dan I believe they didn't care much about this and they'll just build 1.11 with golang 1.5.3 to overcome this. Not sure how they'll fix this in the end. I suppose they'll just force ppl to update their clients.

Comment 6 Daniel Walsh 2016-04-11 19:02:38 UTC
So I guess the question is can we do the same by rhel7.3.

Comment 7 Antonio Murdaca 2016-04-12 08:13:27 UTC
The main problem here is that ppl are shipping docker clients in containers built with go1.4 for instance. Forcing them to upgrade would mean they'll rebuild their images I guess.

Comment 8 Eric Paris 2016-04-12 14:09:27 UTC
@dwalsh while docker might not support customers for long, we do. I do not believe we can or should force our customers to rebuild their images when we know what they messed up.

Comment 9 Antonio Murdaca 2016-04-12 15:24:18 UTC
Eric, Dan, I'll wrote a fix for this at https://github.com/projectatomic/docker/pull/101.
It'll need lots of testing though. The patch is against the upcoming docker-1.11 but we'll probably need this fix in docker-1.10.3 if we're going to build it with go1.6

Comment 10 Antonio Murdaca 2016-05-16 08:16:24 UTC
Our Docker fork already has a patch to fix this BZ, however, I'm working with upstream in order to get the fix there (and we don't carry patches..) https://github.com/docker/docker/pull/22000

Comment 11 Antonio Murdaca 2016-05-20 09:40:08 UTC
The PR upstream has been merged, I've backported the fix to docker-1.10.3 (fedora and rhel) and docker-1.11.

To test it out:

Edit docker.service systemd unit file and add the following env variable:

Environment=DOCKER_HTTP_HOST_COMPAT=1

Reload and restart docker.

Try to contact docker with a docker client <= 1.9.1

Everything works as expected

Comment 12 Daniel Walsh 2016-05-20 13:02:09 UTC
Can't we add something to /etc/sysconfig/docker to do this, rather then editing docker.service systemd unit file.  

I think there is a way to do this also by adding a file to
 /etc/system/systemd/docker.service.d/compat

Environment=DOCKER_HTTP_HOST_COMPAT=1

Comment 13 Antonio Murdaca 2016-05-20 13:04:03 UTC
(In reply to Daniel Walsh from comment #12)
> Can't we add something to /etc/sysconfig/docker to do this, rather then
> editing docker.service systemd unit file.  
> 
> I think there is a way to do this also by adding a file to
>  /etc/system/systemd/docker.service.d/compat
> 
> Environment=DOCKER_HTTP_HOST_COMPAT=1

That's fine as well - though on F24 and F25 I've enabled it by default in systemd unit file (so it's like the patch we were carrying - always enabled)

Comment 14 Daniel Walsh 2016-05-21 11:27:31 UTC
Ok, I guess we can leave this there and then admins can override it, since I guess we want this as the default going forward.

Comment 16 Antonio Murdaca 2016-06-12 17:38:06 UTC
worked with upstream and now the host header overrider is the default behavior so there's not even need to add that ugly env variable.

Comment 17 Luwen Su 2016-06-13 09:04:39 UTC
If the steps is right, then the issue is fine for docker-1.10.3-40.el7.x86_64
#docker -H tcp://10.66.0.137:2376  version
Client:
 Version:         1.9.1
 API version:     1.21
 Package version: docker-common-1.9.1-38.el7.x86_64
 Go version:      go1.4.2
 Git commit:      639e055/1.9.1
 Built:           
 OS/Arch:         linux/amd64

Server:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-40.el7.x86_64
 Go version:      go1.4.2
 Git commit:      7528fb1-unsupported
 Built:           2016-06-09T17:13:37.309698764-04:00
 OS/Arch:         linux/amd64

DEBU[0093] Calling GET /v1.21/version                   
DEBU[0093] GET /v1.21/version                           
DEBU[0093] Warning: client and server don't have the same version (client: 1.9.1, server: 1.10.3) 
DEBU[0093] Unable to read peer creds: server socket is not a Unix socket 
DEBU[0093] {Action=version, LoginUID unknown, PID unknown}

Comment 19 errata-xmlrpc 2016-06-23 16:18:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1274