Bug 1438326

Summary: System-wide "Host *" block incompatible with CanonicalizeHostname directive
Product: [Fedora] Fedora Reporter: Rob Foehl <rwf>
Component: opensshAssignee: Jakub Jelen <jjelen>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 29CC: jjelen, mattias.ellert, mgrepl, plautrba, rh, tmraz
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openssh-7.9p1-4.fc29 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-02-13 02:47:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Rob Foehl 2017-04-03 05:16:09 UTC
Description of problem:
When using the CanonicalizeHostname directive in user ssh config, the system-wide "Host *" section in /etc/ssh/ssh_config.d/05-redhat.conf specifies options that cannot be overridden by other Host sections post-canonicalization, due to the way CanonicalizeHostname is implemented: it re-parses the entire configuration after changing the hostname, and in ssh config, the first matching entry wins.  In the post-canonicalization case, the global "Host *" has already matched, thus options like "GSSAPIAuthentication yes" are forced.


Version-Release number of selected component (if applicable):
openssh-7.4p1-4.fc25.x86_64
openssh-clients-7.4p1-4.fc25.x86_64
openssh-server-7.4p1-4.fc25.x86_64


How reproducible:
100% when using CanonicalizeHostname in user config


Steps to Reproduce:
1. Create a ~/.ssh/config like this:

CanonicalizeHostname yes

Host *.example.com
	GSSAPIAuthentication no

2. Attempt to ssh to a host that would be canonicalized to *.example.com with and without the fully qualified name, and compare the applied config:

╶➤ ssh -G foo |grep gssapiauthentication
gssapiauthentication yes

╶➤ ssh -G foo.example.com |grep gssapiauthentication
gssapiauthentication no


Actual results:
The global "Host *" section wins.


Expected results:
The user configuration wins, post-canonicalization.


Additional info:
This corner case was addressed upstream via the addition of a "Match canonical" directive:

https://lists.mindrot.org/pipermail/openssh-unix-dev/2014-November/033098.html

The syntax changed slightly since that post, and was released in 6.8:

https://bugzilla.mindrot.org/show_bug.cgi?id=2267#c9

Using "Match canonical all" instead of "Host *" in the system-wide configuration should solve this problem.

Comment 1 Jakub Jelen 2017-04-03 06:37:08 UTC
Thank you for the report. Just tested the proposed change and it seems to resolve this issue. I will update Fedora 26 soon

Comment 2 Jakub Jelen 2017-04-03 11:00:15 UTC
Further tests show that just adding "Match canonical all" breaks the connections for servers, where I would expect it would get matched without canonization:

  debug2: checking match for 'canonical all' host x.redhat.com originally x.redhat.com
  debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 8: not matched 'canonical'

To resolve this, we would have to add "CanonicalizeHostname yes", which goes to resolving all the hostnames on the command-line, forces re-reading in all cases:

  debug2: resolve_addr: could not resolve name x.redhat.com as address: Name or service not known
  debug3: resolve_canonicalize: not canonicalizing hostname "x.redhat.com" (max dots 1)
  debug1: Re-reading configuration after hostname canonicalisation

which is most probably something we do not want. I will have a look into that later how it can be solved. Ideas welcomed.

Comment 3 Rob Foehl 2017-04-03 17:42:36 UTC
Hmm, I could've sworn I checked that already...  Sorry, too lost in the details.

For context, this report is the result of giving up after three attempts to come up with a way to make "Host *" in a system config file work in this case without otherwise changing the semantics, which isn't looking likely.  See https://lists.mindrot.org/pipermail/openssh-unix-dev/2017-February/035747.html for the unanswered question which started this.

Unless/until someone can come up with a clever fix upstream, there are two other possibilities specific to the Red Hat packages:

- Remove the "Host *" entirely and avoid setting these options globally

- Avoid shipping a (live) config file, and make the changes in readconf.c:fill_default_options() instead, since they're apparently intended to be defaults

I'd agree that enabling CanonicalizeHostname everywhere is probably not a good idea -- too many surprises.

Comment 4 Jakub Jelen 2017-04-04 07:29:45 UTC
(In reply to Rob Foehl from comment #3)
> - Remove the "Host *" entirely and avoid setting these options globally

This was enabled for years and many people depend on that (as I noticed just few hours after I changed it in my local configuration).

> - Avoid shipping a (live) config file, and make the changes in
> readconf.c:fill_default_options() instead, since they're apparently intended
> to be defaults

This option was most probably not selected because we didn't want to "change default" and make it clear that we use something different than upstream or other distributions. It might be a bit unfortunate, but it sounds like the most viable option for me. But this change should certainly be reflected somehow similar as Ubuntu does it in its manual pages:
http://manpages.ubuntu.com/manpages/zesty/en/man5/ssh_config.5.html
(though they change it the same way as we do)
https://anonscm.debian.org/git/pkg-ssh/openssh.git/tree/ssh_config#n20

We have it this way for more than a decade since RHEL4 or so and there was quite much no problem with that so far.

Petr, Tomas, can you think of any downside of setting this option in code instead of configuration option (though it was still Nalin who intorduced this change)?

Comment 5 Tomas Mraz 2017-04-04 12:04:51 UTC
As you correctly state the reason to set the defaults in the configuration file instead of the code was to minimize the code difference from upstream and to have the downstream change of default properly visible. I'd say the reporter must change the CanonicalizeHostname in a system-wide config file on his own.

Comment 6 Rob Foehl 2017-04-05 05:44:51 UTC
(In reply to Jakub Jelen from comment #4)
> We have it this way for more than a decade since RHEL4 or so and there was
> quite much no problem with that so far.

CanonicalizeHostname has only been around for a couple of years, thus the config re-parsing behavior -- for better or worse -- is significantly newer than this choice of defaults and the use of "Host *" in a system configuration file.

Bug 530741 is one place to start insofar as refuting the claim that these choices never cause problems, even before re-parsing became a thing, and a bit of digging around will turn up a small mountain of evidence that "GSSAPIAuthentication yes" in particular has indeed caused a lot of trouble for people over the years.  Try https://www.google.com/search?q=ssh%20%22GSSAPIAuthentication%22 and https://www.google.com/search?q=site%3Agithub.com%20%22GSSAPIAuthentication%20no%22 for a few glimpses into the scope of the problem.

(In reply to Tomas Mraz from comment #5)
> I'd say the
> reporter must change the CanonicalizeHostname in a system-wide config file
> on his own.

Sigh.  That's not how it works, that isn't relevant to the issue reported here, and what you're actually suggesting -- putting user-specific, host-specific configuration into system config files -- doesn't fly in any real world situations besides one's individual laptop.

Comment 7 Fedora End Of Life 2017-11-16 19:07:50 UTC
This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 8 Rob Foehl 2017-11-27 04:19:53 UTC
This remains an issue in F27, particularly the use of "GSSAPIAuthentication yes".

Comment 9 Ben Cotton 2018-11-27 18:01:40 UTC
This message is a reminder that Fedora 27 is nearing its end of life.
On 2018-Nov-30  Fedora will stop maintaining and issuing updates for
Fedora 27. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora  'version' of '27'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 27 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 10 Ben Cotton 2018-11-30 18:28:26 UTC
Fedora 27 changed to end-of-life (EOL) status on 2018-11-30. Fedora 27 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 11 Rob Foehl 2018-11-30 23:58:43 UTC
Echoing the comment from a year ago, this is still an issue in F28 and F29.  The use of 'Host *' in /etc/ssh/ssh_config.d/05-redhat.conf is not compatible with CanonicalizeHostname.

In addition, the inclusion of /etc/crypto-policies/back-ends/openssh.config by /etc/ssh/ssh_config.d/05-redhat.conf now also means that none of the parameters set therein can be changed by user ssh configuration, either -- regardless of whether CanonicalizeHostname is in use, and especially not on a per-host basis.

The only remaining method to get around either of these is via command line options and ugly shell wrappers, which defeats the purpose of having host-specific config.  There are far too many assumptions being made about the universality of the changes made in the system config...

Comment 12 Jakub Jelen 2018-12-03 10:00:23 UTC
AFAIK, this will be fixed with the last update with the bug #1630166. I will update OpenSSH soon.

Comment 13 Jakub Jelen 2019-01-14 10:47:43 UTC

*** This bug has been marked as a duplicate of bug 1630166 ***

Comment 14 Rob Foehl 2019-01-14 19:12:57 UTC
Bug 1630166 is about the crypto config including directives at the top level, which won't be helped by the upstream change unless all of those directives are moved into a "Match final all" stanza.  Comment 11 mentions this explicitly, and also details why this bug is about the general problem of which 1630166 is a specific instance.

This bug is fundamentally about the use of "Host *" in the distribution-supplied config files.  Including the code to enable "Match final all" by itself doesn't address the issues in the current packaged config for either case.

Comment 15 Fedora Update System 2019-02-06 17:00:52 UTC
openssh-7.9p1-4.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0576ad8060

Comment 16 Fedora Update System 2019-02-10 04:27:33 UTC
openssh-7.9p1-4.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0576ad8060

Comment 17 Fedora Update System 2019-02-13 02:47:11 UTC
openssh-7.9p1-4.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.