Description of problem: When configured for DNS host key verification, ssh relies on DNSSEC to retrieve ssh server key fingerprints. Assuming the use of a properly configured DNS server for both DNSSEC support and SSHFP RR presence, the ssh client retrieves the remote host key fingerprints but cannot certify it was securely obtained from the server without using the EDNS0 extension described in RFC-2671. The Fedora DNS resolver does not currently support an option allowing for EDNS0 to be advertised. OpenBSD seems to have such support (see URL attached). Version-Release number of selected component (if applicable): bind-9.3.2-40.fc6 How reproducible: Always Steps to Reproduce: 1. On ssh server, generate proper SSHFP records with: $ ssh-keygen -r renault-4l -f /etc/ssh/ssh_host_rsa_key.pub renault-4l IN SSHFP 1 1 9900ca808f5ca20daaebbad863127f7eb3d43339 $ ssh-keygen -r renault-4l -f /etc/ssh/ssh_host_dsa_key.pub renault-4l IN SSHFP 2 1 a3c9e96fa6370d114e978ccbc02ae21d947a37a4 2. On DNS server, add the above RR to your zone file, enable DNSSEC, create DNS keys, sign zone file and restart bind. 2. On client side: # cat >> /etc/ssh/ssh_config VerifyHostKeyDNS yes ^D Actual results: $ ssh -v renault-4l.sabre.juniper.net OpenSSH_4.3p2, OpenSSL 0.9.8b 04 May 2006 [...] debug1: found 2 insecure fingerprints in DNS debug1: matching host key fingerprint found in DNS The authenticity of host 'renault-4l.sabre.juniper.net (10.157.12.99)' can't be established. RSA key fingerprint is 83:67:04:09:64:74:1e:b8:25:6a:79:1d:d1:29:83:5c. Matching host key fingerprint found in DNS. Are you sure you want to continue connecting (yes/no)? Expected results: ssh should open a connection to the remote server without asking for fingerprint verification, eventhough no fingerprints for that machine are present in the user's known_hosts file. For this to happen the client resolver needs to use EDNS0 extension which is not currently supported on Fedora. Additional info: This issue might be somewhat related to RHBA-2006:0287-5, although reading the advisory, it is unclear if the fix is on the server side or the resolver side.
Additional information at http://sourceware.org/ml/libc-alpha/2006-09/msg00006.html
Okay, so this part of the problem is in libc, right?
I would assume so. From link in comment #1: "As a workaround one can compile openssh with the resolver library shipped with bind. I tried 9.3.2 myself and it worked (note: you need to set the "edns0" option in /etc/resolv.conf as well). However, in the long term I think that glibc should implement this natively. Maybe this is as easy as re-syncing the resolver in glibc (which seems to have originated from bind) against the latest upstream?"
Try the next rawhide version. Jakub will hopefully update to that version. Note that if ssh directly sets the resolver options you have to recompile the application. If you use resolv.conf you're fine.
BZ requires a separate state change. So, please report success or failure.
Setting to block the generic release notes tracker. We'll monitor this report for content/conclusions to include in the F7 relnotes.
glibc-2.5.90-17 in rawhide should have this code.
I have tried to verify with glibc-2.5.90-17 but could not get it to work. # cat /etc/resolv.conf search sabre.juniper.net options edns0 nameserver 10.157.12.99 $ ssh -v renault-4l.sabre.juniper.net OpenSSH_4.5p1, OpenSSL 0.9.8b 04 May 2006 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug1: Connecting to renault-4l.sabre.juniper.net [10.157.12.99] port 22. debug1: Connection established. debug1: identity file /home/emoret/.ssh/identity type -1 debug1: identity file /home/emoret/.ssh/id_rsa type -1 debug1: identity file /home/emoret/.ssh/id_dsa type -1 debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3 debug1: match: OpenSSH_4.3 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_4.5 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-cbc hmac-md5 none debug1: kex: client->server aes128-cbc hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug1: found 2 insecure fingerprints in DNS debug1: matching host key fingerprint found in DNS The authenticity of host 'renault-4l.sabre.juniper.net (10.157.12.99)' can't be established. Maybe I am not correctly setting the proper options in the resolv.conf file. The resolv.conf man page does not mention the edns0 option. Is there another way to turn that feature on?
Please explain how to set this up. I have added from EDNS0 support what seems to be needed. I will need to reproduce it to see what is missing.
Appart from generating SSHFP resource records and adding them to your dns server as shown in Comment #0, you'll have to sign the zone files on your dns server. Here is how to do it: 1. Enable DNSSEC by adding the following line to the options section of your /etc/named.conf file. dnssec-enable yes; 2. Generate the dns key signing key (KSK) and zone Signing key (ZSK). [root@renault-4l ~]# dnssec-keygen -r /dev/urandom -f KSK -a RSASHA1 -b 1024 -n ZONE sabre.juniper.net Ksabre.juniper.net.+005+65418 [root@renault-4l ~]# dnssec-keygen -r /dev/urandom -a RSASHA1 -b 1024 -n ZONE sabre.juniper.net Ksabre.juniper.net.+005+32690 3. Add the public keys to your zone file: [root@renault-4l ~]# cat >> sabre.juniper.net.db $include Ksabre.juniper.net.+005+32690.key ; ZSK inserted 20040719 $include Ksabre.juniper.net.+005+65418.key ; KSK inserted 20040719 ^D 4. Sign your zone file: [root@renault-4l ~]# dnssec-signzone -r /dev/urandom -o sabre.juniper.net -k Ksabre.juniper.net.+005+65418 /var/named/sabre.juniper.net.db Ksabre.juniper.net.+005+32690.key You should end up with a signed zone file in /var/named/sabre.juniper.net.db.signed. 5. Edit your /etc/named.conf to refer to the signed zone file in place of the unsigned version, then restart your bind server. 6. Make sure your ssh client points to your dnssec server and has the edns0 option enabled.
We were looking to add this to the release notes for test4, but got to this bug too late for that. If the above procedure works, it would be good to make it generic and include it in the final Fedora 7 release notes. To do that, we'd write up a version on: http://fedoraproject.org/wiki/Docs/Beats/Services (or something similar) Is this content accurate? And the feature ready for highlighting?
Don't bother with the release notes, the code needs more work.
Ping, any updates on that ticket?
I've finally set DNS-SEC up on my network and can confirm that ssh doesn't see secure fingerprints. This can be traced back to the reply from the nameserver which doesn't set the AD bit in the reply. The request also doesn't set the AD bit which is the only place I can see libresolv being involved. I don't think the request should have the bit set since this would prevent unsigned responses. It's up to the receiver to filter these out. So my request is: please show me the tcpdump output (or wireshark or whatever) of a successful communication, both the request and the reply. From that I should be able to figure out the next step. So far it seems to me it is either a bind setup problem or a problem is bind.
I don't have a working configuration readily available but I believe OpenBSD supports this feature out of the box. See this post: http://archives.neohapsis.com/archives/openbsd/2005-11/0273.html
Well, I verified that the openbsd ssh behaves identical. I also get the "insecure fingerprints" message which prevents them from automatically be used. This means the glibc code works correct (and has been since I checked in the changes back in February). It is the server setup that is missing something. It might very well be that only keys signed by the upstream organization can be used to successfully authenticate the replies. Anyway, until proven otherwise there I claim there is nothing wrong in libresolv.
The only system i could get this working at the moment was OpenBSD. To enable this i had to provide 'edns0' as an option in resolv.conf[1]. I have attached a PCAP (openbsd.pcap) generated with tcpdump. If you observe it (for instance with Wireshark) you will see that the request for the SSHFP records has the DO bit set in the EDNS0 section of the packet and the response has the AD bit set in the packet header. [1] http://www.mail-archive.com/misc@openbsd.org/msg11176.html
Created attachment 347536 [details] Packet trace of working DNSSEC lookup in OpenBSD
After inspection the main problem is that glibc resolver doesn't set the "DO" bit (RFC 3225) in a request. To be more precise there is no possibility to set the DO bit. DO bit set to "1" is required to get the response with the "AD" bit set. Without DO bit DNSSEC-aware recursive nameservers perform validation (as expected) but they don't return any DNSSEC related data (keys, signatures, and the AD bit set).
Created attachment 355277 [details] proposed patch With this patch (and rebuilt openssh) and "options edns0" in resolv.conf everything works as expected.
One note about "options edns0". In my opinion openssh should be slightly improved as well. Force EDNS0 for all requests is useless and unwise (more bandwidth consumed, possible firewall issues etc).
I committed the patch upstream.
Created attachment 355287 [details] proposed patch for openssh This patch makes SSHFP verifying process working even if "option edns0" is not set in resolv.conf. Tomas, what do you think about this patch, please?
Reassigning to openssh for further inspection.
I would suggest to separate the concerns of DNSSEC and EDNS in the code. EDNS is only needed for DNSSEC if the response exceeds the 512 byte limit of the standard DNS UDP transport. They can be set separately inside of resolv.conf. 'edns0' will only activate the EDNS mechanism to allow for increased packet sizes. 'dnssec' will enforce the DO bit set on requests.
(In reply to comment #25) > I would suggest to separate the concerns of DNSSEC and EDNS in the code. EDNS > is only needed for DNSSEC if the response exceeds the 512 byte limit of the > standard DNS UDP transport. > > They can be set separately inside of resolv.conf. 'edns0' will only activate > the EDNS mechanism to allow for increased packet sizes. 'dnssec' will enforce > the DO bit set on requests. RFC 4035 clearly specifies relationship between DNSSEC and EDNS0: -------- 4.1. EDNS Support A security-aware resolver MUST include an EDNS ([RFC2671]) OPT pseudo-RR with the DO ([RFC3225]) bit set when sending queries. A security-aware resolver MUST support a message size of at least 1220 octets, SHOULD support a message size of 4000 octets, and MUST use the "sender's UDP payload size" field in the EDNS OPT pseudo-RR to advertise the message size that it is willing to accept. A security-aware resolver's IP layer MUST handle fragmented UDP packets correctly regardless of whether any such fragmented packets were received via IPv4 or IPv6. Please see [RFC1122], [RFC2460], and [RFC3226] for discussion of these requirements. --------- As you can see if you would like to use DNSSEC you have to use EDNS0 in both request and response. If you would like to save bandwidth by setting EDNS0 payload size lesser than 4000 octets (to 1220 octets, for example) it is not a win because recursive server will respond with "TC" bit set thus you have to reconnect to the server via TCP and fetch the entire response. If response is shorter than 4000 octets then, of course, EDNS0 packet is shortened appropriately.
My suggestion however was to consider _not_ to send the DO bit when EDNS is enabled since EDNS is not directly dependent on it. The other way around i totally agree. With the current implementation as far as i understand it it is not possible to use EDNS without using the DO bit and thus DNSSEC.
(In reply to comment #27) > My suggestion however was to consider _not_ to send the DO bit when EDNS is > enabled since EDNS is not directly dependent on it. The other way around i > totally agree. With the current implementation as far as i understand it it is > not possible to use EDNS without using the DO bit and thus DNSSEC. Ah, sorry, I didn't understand correctly your comment #25. In glibc code EDNS0 and DNSSEC are completely separated if I read the code correctly. If you set the "RES_USE_EDNS0" flag (in resolver->options field) then requiest includes OPT pseudo RR (without DO bit set). If you set the RES_USE_DNSSEC flag then request contains OPT pseudo RR with DO bit set. The "edns0' option in resonv.conf doesn't set the DO bit in request. I don't see any benefit from new "dnssec" option because none of current application checks the AD flag in response. And if app would like to check it then it will simply set RES_USE_DNSSEC flag. But final decision depends on Ulrich.
Ok, then i misunderstood your initial comments about the implementation. :) I thought that GLIBC is now setting the DO bit once 'edns0' has been enabled in resolv.conf. If that is still up to the application it should be fine.
I believe this may still not be fixed. I have setup my dns and ssh servers on a CentOS5 host and my client runs rawhide. Here is the content of my client's resolv.conf file: $ cat resolv.conf # Generated by NetworkManager options edns0 domain zouric.com search zouric.com nameserver 192.168.1.133 A DNS trace: # tcpdump -i eth0 -vvv dst 192.168.1.133 and port 53 22:32:13.269475 IP (tos 0x0, ttl 64, id 36922, offset 0, flags [DF], proto UDP (17), length 75) 192.168.1.54.42314 > tracore.ozon-e.com.domain: [bad udp cksum e6cd!] 60986+ [1au] A? tracore.zouric.com. ar: . OPT UDPsize=1024 (47) 22:32:13.274100 IP (tos 0x0, ttl 64, id 36927, offset 0, flags [DF], proto UDP (17), length 83) 192.168.1.54.44981 > tracore.ozon-e.com.domain: [bad udp cksum 4d4c!] 4124+ [1au] PTR? 133.1.168.192.in-addr.arpa. ar: . OPT UDPsize=1024 (55) 22:32:13.277227 IP (tos 0x0, ttl 64, id 36930, offset 0, flags [DF], proto UDP (17), length 82) 192.168.1.54.39307 > tracore.ozon-e.com.domain: [bad udp cksum 7cfd!] 56034+ [1au] PTR? 54.1.168.192.in-addr.arpa. ar: . OPT UDPsize=1024 (54) 22:32:13.412265 IP (tos 0x0, ttl 64, id 37065, offset 0, flags [DF], proto UDP (17), length 75) 192.168.1.54.47117 > tracore.ozon-e.com.domain: [bad udp cksum 35a0!] 2178+ [1au] SSHFP? tracore.zouric.com. ar: . OPT UDPsize=65535 OK (47) The ssh command: # ssh -v -4 tracore.zouric.com OpenSSH_5.2p1, OpenSSL 1.0.0-fips-beta3 15 Jul 2009 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug1: Connecting to tracore.zouric.com [192.168.1.133] port 22. debug1: Connection established. debug1: permanently_set_uid: 0/0 debug1: identity file /root/.ssh/identity type -1 debug1: identity file /root/.ssh/id_rsa type -1 debug1: identity file /root/.ssh/id_dsa type -1 debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3 debug1: match: OpenSSH_4.3 pat OpenSSH_4* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_5.2 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-ctr hmac-md5 none debug1: kex: client->server aes128-ctr hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY debug1: found 2 insecure fingerprints in DNS debug1: matching host key fingerprint found in DNS debug1: checking without port identifier The authenticity of host '[tracore.zouric.com]:22 ([192.168.1.133]:22)' can't be established. RSA key fingerprint is b2:38:99:62:29:4c:99:b1:61:8c:85:e1:11:11:61:ad. Matching host key fingerprint found in DNS. Are you sure you want to continue connecting (yes/no)? ^C
(In reply to comment #30) > I believe this may still not be fixed. I have setup my dns and ssh servers on a > CentOS5 host and my client runs rawhide. Here is the content of my client's > resolv.conf file: > $ cat resolv.conf > # Generated by NetworkManager > options edns0 > domain zouric.com > search zouric.com > nameserver 192.168.1.133 Is 192.168.1.133 server authoritative for zouric.com? If yes then SSHFP verification can't work because authoritative response != validated response. You have to setup additional DNS server (DNSSEC validator) which will validate zouric.com zone records: ---------- -------------------- ---------- | zouric.com | -> | recursive nameserver | -> | ssh client | | nameserver | -------------------- ---------- ---------- You have to sign zouric.com zone, then put appropriate zone signing key to recursive nameserver's trust anchors and use recursive server as nameserver in /etc/resolv.conf. > A DNS trace: > # tcpdump -i eth0 -vvv dst 192.168.1.133 and port 53 The `dig` utility is far more better for DNS debugging than tcpdump. Would it be possible to attach output from `dig @192.168.1.133 tracore.zouric.com SSHFP +dnssec`, please?
Here is a test system with the same results: dig @68.87.69.154 eagle.roysdon.net SSHFP +dnssec dig @68.87.68.170 eagle.roysdon.net SSHFP +dnssec roysdon.net is DNSSEC signed and in dlv.isc.org. 68.87.69.154 is Comcast's Unbound DLV-enabled test server (bvrt-dnssec-trial.beaverton.or.bverton.comcast.net.) and 68.87.68.170 is Comcast's ISC Bind DLV-enabled test server (atlt-dnssec-trial.s3woodstock.ga.atlanta.comcast.net). Both responses have the AD flag in the response showing DNSSEC signatures are coming back verified. However, I still get "found 2 insecure fingerprints in DNS" with: ssh -vv -o 'VerifyHostKeyDNS yes' eagle.roysdon.net This is on a current F11 system (openssh-5.2p1-2.fc11.i586) with /etc/resolv.conf: nameserver 68.87.69.154 options edns0
This bug has been fixed in rawhide (aka development tree). It means it will work in Fedora 12, not in Fedora 11.
Tried again with F12-Alpha and it works just fine. -o 'VerifyHostKeyDNS yes' or VerifyHostKeyDNS yes in /etc/ssh/ssh_config causes no prompting for a new SSH host public key which SSHFP matches from a DNSSEC-enabled zone on a DNSSEC resolver.
This bug appears to have been reported against 'rawhide' during the Fedora 12 development cycle. Changing version to '12'. More information and reason for this action is here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
This message is a reminder that Fedora 12 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 12. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '12'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 12's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 12 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Fedora 12 changed to end-of-life (EOL) status on 2010-12-02. Fedora 12 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. Thank you for reporting this bug and we are sorry it could not be fixed.